Operation video processing method and device based on multi-modal large model
Through the multimodal large model combining video encoder and large language model, surgical videos are analyzed and natural language Q&A supports, the problem of poor interactiveness of surgical video understanding methods in the existing technology is solved, and more efficient surgical video analysis and real-time feedback are achieved. .
Patent Information
- Application Number
- CN202410894276.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-04
AI Technical Summary
The existing surgical video understanding methods are poorly interactive and less practical, unable to effectively support natural language communication between doctors and systems, and perform poorly during multitasking.
Using a surgical video processing method based on a multimodal large model, the surgical video is analyzed and natural language question-and-answer is supported by combining a video encoder and a large language model. The surgical video is encoded and processed through a pre-trained video encoder to convert the features to match. Multimodal large model space and cross-embedded with text description to generate text answer content.
It enhances the interactiveness and versatility of surgical video understanding, provides real-time feedback and detailed analysis, improves surgical efficiency and patient treatment effects, and supports doctors to query surgical progress through natural language.
Smart Images

Figure CN119048947A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a surgical video processing method and device based on a multimodal large model. Background Art
[0002] In today's medical field, with the rapid development of computer technology, the analysis and understanding of surgical videos has become a key technology to improve surgical accuracy and patient safety.
[0003] However, although existing surgical video understanding algorithms have assisted the surgical process to some extent, these algorithms mainly target single tasks such as stage recognition or tool detection, and often only provide limited outputs and lack interactivity with doctors.
[0004] This shows that the surgical video understanding methods in related technologies have technical problems of poor interactivity and low practicality. Summary of the invention
[0005] The present invention provides a surgical video processing method and device based on a multimodal large model, so as to solve the defects of poor interactivity and low practicality of the surgical video understanding method in the prior art, so as to support multi-task question answering based on natural language while analyzing the surgical video.
[0006] The present invention provides a surgical video processing method based on a multimodal large model, comprising the following steps: determining a surgical video and an original question related to the surgical video; splitting the surgical video to obtain a plurality of video segments with a fixed number of frames; encoding each of the plurality of video segments through a pre-trained video encoder to obtain an abstract feature; converting the spatial dimension of the abstract feature to be consistent with the spatial dimension of a preset multimodal large model through a preset multimodal converter to obtain a processed abstract feature; cross-embedding the processed abstract feature with a text-assisted description to obtain a mixed abstract feature; inputting the mixed abstract feature and the original question into the preset multimodal large model to obtain a text answer content output by the preset multimodal large model.
[0007] According to a surgical video processing method based on a multimodal large model provided by the present invention, before determining the surgical video and the original problem related to the surgical video, the method also includes: training a preset video encoder to obtain a pre-trained video encoder, wherein the pre-trained video encoder is used to align surgical instrument video features with surgical instrument text features.
[0008] According to a surgical video processing method based on a multimodal large model provided by the present invention, the training of a preset video encoder to obtain a pre-trained video encoder includes: obtaining a surgical video training sample, wherein the surgical video training sample includes at least one surgical instrument mark and a surgical instrument text feature corresponding to the surgical instrument mark; performing instrument masking on the surgical video training sample by a preset video encoder to obtain a masked video training sample; encoding the masked video training sample by an encoder of the preset video encoder to obtain an encoded video training sample; performing mask reshaping on the encoded video training sample by a decoder of the preset video encoder to obtain a surgical instrument video feature; and updating the preset video encoder based on a preset loss function between the surgical instrument video feature and the surgical instrument text feature to obtain a pre-trained video encoder.
[0009] According to a surgical video processing method based on a multimodal large model provided by the present invention, the surgical video is split to obtain multiple video segments with a fixed number of frames, including: converting the surgical video into continuous image frames; according to a preset target number of frames, converting the continuous image frames into multiple video segments, wherein the number of frames of the video segments is the preset target number of frames.
[0010] According to a surgical video processing method based on a multimodal large model provided by the present invention, the abstract features after processing are cross-embedded with text-assisted descriptions to obtain mixed abstract features, including: determining a timeline segment of the video segment corresponding to the processed abstract features; generating a text-assisted description based on the timeline segment according to a preset text description template; converting the text-assisted description through a preset text token processor to obtain text abstract features; and cross-embedding the processed abstract features with the text abstract features to obtain mixed abstract features.
[0011] According to a surgical video processing method based on a multimodal large model provided by the present invention, the mixed abstract features and the original question are input into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model, including: determining the question type of the original question; calling the preset multimodal large model to determine the target pre-trained low-rank adaptation module based on the question type; and outputting the text answer content based on the target pre-trained low-rank adaptation module according to the mixed abstract features and the original question.
[0012] The present invention also provides a surgical video processing device based on a multimodal large model, comprising the following modules: a determination module, used to determine a surgical video and an original question related to the surgical video; a splitting module, used to split the surgical video to obtain multiple video segments with a fixed number of frames; an encoding module, used to encode each of the multiple video segments through a pre-trained video encoder to obtain an abstract feature; a conversion module, used to convert the spatial dimension of the abstract feature to be consistent with the spatial dimension of a preset multimodal large model through a preset multimodal converter to obtain a processed abstract feature; an embedding module, used to cross-embed the processed abstract feature with a text-assisted description to obtain a mixed abstract feature; an output module, used to input the mixed abstract feature and the original question into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model.
[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the surgical video processing method based on a multimodal large model as described above is implemented.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described surgical video processing methods based on a multimodal large model.
[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned surgical video processing methods based on a multimodal large model.
[0016] The surgical video processing method and device based on the multimodal large model provided by the present invention determine the surgical video and the original question related to the surgical video so as to associate the corresponding video segment according to the original question in the future; split the surgical video to obtain multiple video segments with a fixed number of frames; encode each video segment in the multiple video segments through a pre-trained video encoder to obtain an abstract feature, and convert the spatial dimension of the abstract feature to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter to obtain the processed abstract feature; cross-embed the processed abstract feature with the text auxiliary description to obtain a mixed abstract feature; input the mixed abstract feature and the original question into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model, thereby combining the video encoder and the multimodal large model, and being able to support multi-task question and answer based on natural language while parsing the surgical video; in this way, the model can be directly queried with natural language to obtain real-time feedback and detailed analysis on the progress of the operation, which greatly enhances the interactivity and versatility of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 It is a flow chart of the surgical video processing method based on the multimodal large model provided by the present invention.
[0019] Figure 2 This is a flowchart of surgical video understanding based on a multimodal large model provided by the present invention.
[0020] Figure 3 This is a flow chart of stage one provided by the present invention: surgical instrument mask pre-training.
[0021] Figure 4 This is a flow chart of stage 2 provided by the present invention: alignment of text descriptions of surgical videos.
[0022] Figure 5 This is a flowchart of stage three provided by the present invention: multimodal large model training.
[0023] Figure 6 It is a structural schematic diagram of a surgical video processing device based on a multimodal large model provided by the present invention.
[0024] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0026] In today's medical field, with the rapid development of computer technology, the analysis and understanding of surgical videos has become a key technology to improve surgical accuracy and patient safety. Multimodal large models, by combining video analysis and natural language processing technology, provide an unprecedented new way to support surgical decision-making. These models can not only analyze surgical videos, but also optimize surgical results and improve operational efficiency by understanding the multi-dimensional information of surgical scenes.
[0027] However, although existing surgical video understanding algorithms have assisted the surgical process to some extent, these algorithms mainly focus on single tasks, such as stage recognition or tool detection, and often only provide limited outputs and lack interactivity with doctors. Doctors cannot effectively communicate with these models through natural language to obtain deep insights and answers to specific clinical situations. In addition, these algorithms usually perform poorly when handling combined multi-tasks, such as simultaneous surgical tool detection and stage recognition, further limiting their practicality and flexibility.
[0028] In response to these challenges, the present invention proposes a surgical video processing method based on a multimodal large model. This method combines a video encoder and a large language model, so that it can support multi-task question and answer based on natural language while parsing surgical videos. In this way, doctors can directly query the model in natural language to obtain real-time feedback and detailed analysis of the progress of the operation, which greatly enhances the interactivity and versatility of the model. In addition, the present invention also takes into account the complexity and dynamic changes of the surgical scene. Through deep learning and pattern recognition technology, it can accurately identify and respond to doctors' query needs in a changing surgical environment. The present invention not only improves the accuracy of surgical video understanding, but also provides a powerful decision support tool for the surgical team, thereby achieving higher surgical efficiency and better patient treatment effects.
[0029] The purpose of the present invention is to solve the problem that the existing surgical video analysis and understanding functions are single and the interactivity is poor, so as to enhance the interactivity and versatility of the model, improve surgical efficiency and achieve better patient treatment results.
[0030] In order to achieve the above objectives, the present invention provides a surgical video processing method based on a multimodal large model, which uses a video encoder and a language large model to deeply understand and analyze the surgical video.
[0031] refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a surgical video processing method based on a multimodal large model provided by the present invention. Figure 1 As shown, the method includes the following: Step 101, determining a surgical video and an original question related to the surgical video.
[0032] In an embodiment of the present invention, the surgical video may be video data transmitted in real time or video data of a pre-stored surgical video; and the original question related to the surgical video may be a multi-task question and answer based on natural language by the user (doctor), for example, what surgical instrument should be used in the current surgical stage, what medicine should be selected according to the current patient condition, etc., which is not limited in the present invention.
[0033] In an embodiment of the present invention, after obtaining the original question related to the surgical video, the original question is preprocessed to obtain a text vector that can be recognized by a preset multimodal large model.
[0034] For example, the text of the original question is divided into multiple words or tags (removing common words such as modal particles that are not very helpful in understanding the meaning of the text); multiple words are embedded through a preset word embedding model (for example, Word2Vec, etc.) to obtain a text sequence (positional encoding is added to retain the order information of the words in the sequence); the processed word embedding sequence (also including other types of embedding, such as paragraph embedding, position embedding, etc.) is passed to the pre-trained video encoder and the preset multimodal large model.
[0035] Step 102, split the surgical video into multiple video segments with a fixed number of frames.
[0036] In an embodiment of the present invention, the surgical video is converted into continuous image frames; according to a preset target frame number, the continuous image frames are converted into a plurality of video segments, wherein the frame number of the video segments is the preset target frame number.
[0037] For example, the surgical video V is divided into multiple shorter video segments {v1, v2, v3…, v n}, where n represents the sequence number of the video segment (the total number of video segments).
[0038] Step 103: encode each of the multiple video segments using a pre-trained video encoder to obtain an abstract feature.
[0039] In the embodiment of the present invention, continuous image frames are converted into shorter video segments containing fewer image frames according to a specific number of frames, and each shorter video segment is passed through a pre-trained video encoder to obtain an abstract feature.
[0040] Here, the pre-trained video encoder is a model that is trained on a large-scale dataset to learn a general representation of video content, so as to be able to handle various video understanding tasks. In an embodiment of the present invention, the pre-trained video encoder is used to solve a wide range of video understanding tasks, including classification, positioning, retrieval, subtitle generation, and question answering.
[0041] In the embodiment of the present invention, the pre-trained video encoder is pre-trained using a two-stage training method, wherein the first stage is contrast learning, and the second stage is predicting occluded video blocks.
[0042] In contrastive learning, the video encoder is aligned with the text encoder using all video-text pairs. The symmetric cross entropy loss is minimized by minimizing the similarity scores of all video-text pairs in the batch.
[0043] In predicting masked video patches, we predict first-stage video-level global and tokenized embeddings from unmasked input video patches. The encoder output tokens are randomly shuffled before passing to the decoder to avoid learning shortcuts. We leverage extensive video-only data to refine semantic embeddings globally and token-wise for improved masked video modeling.
[0044] Step 104 , converting the spatial dimension of the abstract feature to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter, thereby obtaining the processed abstract feature.
[0045] In an embodiment of the present invention, the preset multimodal transformer may be a Q-Former (Querying Transformer), which is used for efficient interaction and fusion between vision and language models.
[0046] The abstract features are passed through Q-Former to obtain Q={q1,q2,q3,…,q n}, so that its spatial dimension is consistent with the spatial dimension of the multimodal large model. That is, Q-Former uses a learnable query vector set (denoted as Q={q1, q2,..., q n}, where n is the vector number) extracts visual features from the visual model and transforms these features into representations consistent with the spatial dimensions of the preset multimodal large model.
[0047] Through the above steps, Q-Former is able to transform abstract visual features (represented by query vectors) into representations that are consistent with the spatial dimensions of the multimodal large model. This representation not only helps the large language model understand and generate text related to visual content, but also improves the overall performance and generalization ability of the multimodal learning model.
[0048] Step 105 , cross-embedding is performed based on the processed abstract features and the text-assisted description to obtain mixed abstract features.
[0049] In the embodiment of the present invention, the text-assisted description is used to describe the start time and the end time of the video segment corresponding to the processed abstract feature, for example, this is a video segment from the 0th to the dth second.
[0050] In an embodiment of the present invention, a timeline segment of a video segment (in a surgical video) corresponding to a processed abstract feature is determined; a text-assisted description is generated based on the timeline segment according to a preset text description template; the text-assisted description is converted by a preset text token processor to obtain a text abstract feature; and a mixed abstract feature is obtained by cross-embedding the processed abstract feature with the text abstract feature.
[0051] Step 106, input the mixed abstract features and the original question into a preset multimodal large model to obtain a text answer content output by the preset multimodal large model.
[0052] In an embodiment of the present invention, the preset multimodality obtains mixed abstract features and original questions. For different types of questions, the preset multimodality will select the LoRA module trained accordingly according to its type, so that the answer to the question is more targeted.
[0053] Here, the LoRA module (Low-Rank Adaptation of Large Language Models) is a low-rank parameter fine-tuning technology for large language models (LLMs). By training a low-rank matrix and injecting its parameters into the large language model, customized adjustments to the model's generation style or specific task capabilities can be achieved.
[0054] Through the above steps of the embodiment of the present invention, the surgical video and the original question related to the surgical video are determined; the surgical video is split to obtain multiple video segments with a fixed number of frames; each of the multiple video segments is encoded by a pre-trained video encoder to obtain abstract features; the spatial dimension of the abstract feature is converted to be consistent with the spatial dimension of a preset multimodal large model by a preset multimodal converter to obtain processed abstract features; the processed abstract feature is cross-embedded with the text-assisted description to obtain a mixed abstract feature; the mixed abstract feature and the original question are input into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model; thereby solving the technical problems of poor interactivity and low practicality in the surgical video understanding method in the related art.
[0055] According to a surgical video processing method based on a multimodal large model provided by the present invention, before determining the surgical video and the original problem related to the surgical video, the method further includes: A preset video encoder is trained to obtain a pre-trained video encoder, wherein the pre-trained video encoder is used to align surgical instrument video features with surgical instrument text features.
[0056] In an embodiment of the present invention, the pre-trained video encoder V e , a mask is generated for the position of the instrument, and the video content is read into the encoder through the auto encoder, and the mask of the position of the instrument is reshaped through the decoder. After that, the text segment matching the corresponding text description is aligned on the auto encoder.
[0057] According to a surgical video processing method based on a multimodal large model provided by the present invention, a preset video encoder is trained to obtain a pre-trained video encoder, comprising: Acquire a surgical video training sample, wherein the surgical video training sample includes at least one surgical instrument mark and a surgical instrument text feature corresponding to the surgical instrument mark; Performing instrument masking on surgical video training samples through a preset video encoder to obtain masked video training samples; Encoding the masked video training sample by an encoder of a preset video encoder to obtain an encoded video training sample; Masking and reshaping the encoded video training samples through a decoder of a preset video encoder to obtain surgical instrument video features; Based on a preset loss function between the video features of the surgical instrument and the text features of the surgical instrument, the preset video encoder is updated to obtain a pre-trained video encoder.
[0058] In an embodiment of the present invention, a pre-trained video encoder is used to identify specific objects (such as the location of a medical device) in a video, and to generate a mask of the specific object through an autoencoder structure, and further to align the video content with the text description.
[0059] In an embodiment of the present invention, video data containing target medical devices is collected. The video content is ensured to be diverse to cover different devices, positions, and angles. Each frame in the video data is annotated to clearly mark the location of the medical device (completed by bounding boxes or pixel-level masks). A detailed text description is provided for each video segment or key frame in the video, and the description should include information such as the name, location, and action of the device to obtain surgical video training samples.
[0060] The preset video encoder includes an encoder part and a decoder part. The pre-trained video encoder is used as the encoder part of the autoencoder to extract features from the input video. The decoder part structure should match the encoder but in the opposite direction, and is used to reconstruct video frames from the encoded features or generate a mask of the device position.
[0061] Based on a preset loss function (e.g., a cross entropy loss function) between the video features of surgical instruments and the text features of surgical instruments, an annotated video dataset is used to train the autoencoder, and the parameters of the encoder and decoder are optimized through a back-propagation algorithm to obtain a pre-trained video encoder.
[0062] According to a surgical video processing method based on a multimodal large model provided by the present invention, the surgical video is split to obtain multiple video segments with a fixed number of frames, including: Convert surgical videos into sequential image frames; According to a preset target frame number, continuous image frames are converted into a plurality of video segments, wherein the frame number of the video segment is the preset target frame number.
[0063] In the embodiment of the present invention, the surgical video V is divided into a plurality of shorter video segments {v1, v2, v3…, v n}, where n represents the total number of video segments, and video segment v of surgical video V n Contains a series of consecutive image frames {x1,x2,x3,…,x m}, m represents the total number of frames (the preset target number of frames), where x t ∈ R H×W×M , each x t Represents a frame in the video, H, W, and M represent the height, width, and number of color channels of the image, respectively (for RGB images, M=3).
[0064] According to the specific frame number d, the continuous image frames {x1,x2,x3,…,x m} into a shorter video segment v1={x1,x2,x3,…x d},v2={x d+1 ,x d+2 ,x d+3 ,…x d+d-1},…vn={x (n-1)*d+1 , x (n-1)*d+2 ,x (n-1)*d+3, …, x n*d-1}.
[0065] The video segment of the surgical video can be expressed by the following formula (1): (1)
[0066] in, Surgery video The Video segments, Represents a video frame, Indicates the sequence number of the video segment. Indicates the sequence number of the video frame, Indicates the preset target frame number.
[0067] After obtaining a plurality of video segments with a fixed number of frames, the method further includes: Each shorter video segment Through the video encoder Then we get the abstract feature E = {e1,e2,e3,…,e n}.
[0068] Each feature in the abstract feature can be expressed by the following formula (2): (2)
[0069] in, Represents the first Features, Surgery video The Video segments; Indicates the sequence number of the video segment.
[0070] The abstract features are passed through Q-Former to obtain Q={q1,q2,q3,…,q n}, so that its spatial dimension is consistent with the spatial dimension of the large model (multimodal large model), which can be expressed by the following formula (3): (3)
[0071] in, represents the processed abstract features, Represents the first Features, represents the preset multimodal converter, Represents the first Features.
[0072] According to a surgical video processing method based on a multimodal large model provided by the present invention, a hybrid abstract feature is obtained by cross-embedding the processed abstract features with the text-assisted description, including: Determine a timeline segment of the video segment corresponding to the processed abstract feature; Generate text-assisted descriptions based on the timeline paragraphs according to the preset text description template; The text-assisted description is converted by a preset text token processor to obtain text abstract features; The processed abstract features are cross-embedded with the text abstract features to obtain mixed abstract features.
[0073] In the embodiment of the present invention, the processed abstract features are combined with the built-in text auxiliary description segment T default Cross embedding is used to obtain the mixed abstract feature M as the input of the large model. The specific process is as follows: First, add matching text before each short video segment, for example: {"This is a video clip from 0 to d seconds", q1; "This is a video clip from d+1 to 2d-1 seconds", q2; "This is a video clip from 2d to 3d-1 seconds", q3; ...; "This is a video clip from (n-1)*d to n*d-1 seconds", q n}.
[0074] Here, each word is converted into a word abstract feature by a word token processor, and finally mixed to obtain a mixed abstract feature M={M1,M2,M3,…,M n}, which can be specifically expressed by the following formula (4): (4)
[0075] in, Represents the first Features, It indicates that the preset text token processor is used to cross-embed the processed abstract features with the text abstract features. Indicates Text abstract features, Indicates The processed abstract features, Represents the index, Indicates the total number of indexes.
[0076] According to a surgical video processing method based on a multimodal large model provided by the present invention, the mixed abstract features and the original question are input into a preset multimodal large model to obtain text answer content output by the preset multimodal large model, including: Determine the problem type of the original question; Call the preset multimodal large model based on the problem type and determine the target pre-trained low-rank adaptation module; The target pre-trained low-rank adaptation module outputs textual answers based on the mixed abstract features and the original question.
[0077] In the embodiment of the present invention, the preset multimodal large model obtains the mixed abstract feature M={M1,M2,M3,…,M n} and the original question T; for different types of questions, the big model will select the corresponding trained LoRA module according to its type, so that the response to the question is more targeted; the preset multimodal big model receives the mixed abstract feature M and the original question T raised by the user as input, and outputs the text answer content.
[0078] When dealing with different types of questions, the preset multimodal large model can select the corresponding, trained LoRA module based on the type or field of the question, where each LoRA module is trained for questions of a specific type or field, so they can capture the language patterns and knowledge unique to that type of question, thereby generating more accurate and targeted responses. By training multiple LoRA modules for different tasks or fields, the preset multimodal large model can easily expand its capabilities without retraining the entire model. This allows the model to quickly adapt to new scenarios and needs.
[0079] refer to Figure 2 , Figure 2 The present invention provides a flowchart for understanding surgical videos based on a multimodal large model, which specifically includes the following steps.
[0080] Step 201, pre-training a video encoder.
[0081] Step 202, receiving video and text commands.
[0082] Step 203, cutting the video into a set of shorter segments and passing them through a pre-trained video encoder.
[0083] Step 204: extracting video abstract features through Q-Former.
[0084] Step 205: Mix and embed the extracted video abstract features with the text-assisted description.
[0085] Step 206, input the text commands and mixed embedding features of the model into the large model.
[0086] In step 207, the large model generates an answer.
[0087] Through the embodiments provided by the present invention, a multimodal large model is combined with video coding technology to improve the depth and breadth of surgical video understanding. By segmenting and converting continuous surgical videos into abstract features, combined with the deep learning capabilities of the large model, a comprehensive analysis and real-time interaction of the surgical process can be achieved. The surgical instrument mask pre-training proposed in the present invention can ensure that the abstract feature extraction of the video surgical video can focus mainly on the changes and movements of the instruments. In addition, through the cross-embedding of built-in text descriptions and video features, the model's semantic understanding ability of surgical videos is enhanced, so that the model can not only answer queries about the surgical stage, but also analyze surgical tools and operation details.
[0088] The model of the surgical video processing method based on the multimodal large model provided by the present invention includes three main network modules, namely a video encoder, a Q-Former, and a multi-expert mixed fine-tuning large language model. Here, the first module is a video encoder, which takes the surgical video as input and compresses and extracts its visual content. The second module is Q-Former, which is used to align the input space of the video encoder and the input space of the large model. The third module is a multi-expert mixed fine-tuning large language model, which is used to answer the doctor's questions based on the video content and answer them in natural language.
[0089] refer to Figure 3 , Figure 3 It is a schematic diagram of the process of stage one: surgical instrument mask pre-training provided by the present invention, including inputting a surgical video (a series of image frames), mask (instrument mask) processing, encoding by an encoder, obtaining encoded video training samples, decoding by a decoder, and obtaining surgical instrument video features.
[0090] refer to Figure 4 , Figure 4 It is a schematic diagram of the process of stage 2: alignment of text descriptions of surgical videos provided by the present invention, including text input, a pre-trained language representation model (BERT) for receiving the text input, video input, an encoder for receiving the video input, and contrast loss.
[0091] refer to Figure 5 , Figure 5It is a schematic diagram of the process of stage three: multimodal large model training provided by the present invention, including segmenting the video input into multiple segments (including segment 1 and segment 2), a pre-trained video encoder (encoder), the original question, a gated unit network, Qformer, abstract feature 1, abstract feature 2, text-assisted embedding, LoRA modules (LoRA1, LoRA2, LoRA3, LoRA4), a language large model, and text output.
[0092] In practical applications, taking laparoscopic cholecystectomy endoscopic video as an example, the embodiment of the present invention specifically includes the following steps: Step S0, for laparoscopic cholecystectomy endoscopic video to video encoder V e Pre-training is performed to generate a mask for the device position, and the video content is read into the encoder through the autoencoder, and the mask for the device position is reshaped through the decoder. After that, the text segment matching the autoencoder with the corresponding text description is aligned, and the model parameters are updated by calculating the cross entropy loss function between the video features and the text features.
[0093] Step S1, receiving the laparoscopic cholecystectomy endoscopic video V and the doctor's specific questions T
[0094] Step S2, dividing the laparoscopic cholecystectomy endoscopic video V into a plurality of shorter video segments {v1, v2, v3…, v n}, each shorter video segment passes through the video encoder V e Get the abstract features {e1,e2,e3,…,e n}, the specific process is as follows: First, the laparoscopic cholecystectomy endoscopic video V contains a series of consecutive image frames {x1, x2, x3, …, x m}, m represents the total number of frames (the preset target number of frames), where x t ∈ R H×W×M , each x t Represents a frame in the video, H, W, and M represent the height, width, and number of color channels of the image, respectively (for RGB images, M=3).
[0095] Secondly, according to the specific frame number d, the consecutive image frames {x1,x2,x3,…,x m} into a shorter video segment v1={x1,x2,x3,…x d},v2={x d+1 ,x d+2 ,x d+3 ,…x d+d-1},…vn={x (n-1)*d+1 ,x (n-1)*d+2 , x (n-1)*d+3, …, x n*d-1}.
[0096] The video segment of the surgical video can be expressed by the following formula (5): (5)
[0097] Finally, each shorter video segment v passes through the video encoder V e Then we get the abstract feature E = {e1,e2,e3,…,e n}.
[0098] Each feature in the abstract feature can be expressed by the following formula (6): (6)
[0099] in, Represents the first Features, Surgery video The Video segments; Indicates the sequence number of the video segment.
[0100] Step S3: The abstract features are passed through Q-Former to obtain Q={q1,q2,q3,…,q n}, so that its spatial dimension is consistent with the spatial dimension of the large model, which can be expressed by the following formula (7): (7)
[0101] in, represents the processed abstract features, Represents the first Features, represents the preset multimodal converter, Represents the first Features.
[0102] Step S4: combine the processed abstract features with the built-in text auxiliary description segment T default Cross embedding is used to obtain the mixed abstract feature M as the input of the large model. The specific process is as follows: First, add matching text before each short video segment, for example: {"This is a video clip from 0 to d seconds", q1; "This is a video clip from d+1 to 2d-1 seconds", q2; "This is a video clip from 2d to 3d-1 seconds", q3; ...; "This is a video clip from (n-1)*d to n*d-1 seconds", q n}.
[0103] Each word will be converted into a word abstract feature by the word token processor, and finally mixed to obtain a mixed abstract feature M={M1,M2,M3,…,M n}, which can be specifically expressed by the following formula (8): (8)
[0104] in, Represents the first Features, It indicates that the preset text token processor is used to cross-embed the processed abstract features with the text abstract features. Indicates Text abstract features, Indicates After processing, the abstract features Represents the index, Indicates the total number of indexes.
[0105] Step S5: The large model obtains the mixed abstract features M={M1,M2,M3,…,M n} and the original question T.
[0106] Step S6: For different types of questions, the large model will select the corresponding trained LoRA module according to its type, so that the answer to the question is more targeted.
[0107] Step S6, the large model receives M and the doctor's question T as input and outputs a text answer.
[0108] The surgical video processing device based on a multimodal large model provided by the present invention is described below. The surgical video processing device based on a multimodal large model described below and the surgical video processing method based on a multimodal large model described above can be referenced to each other.
[0109] refer to Figure 6 , Figure 6 It is a structural schematic diagram of a surgical video processing device based on a multimodal large model provided by the present invention, including a determination module 601, a splitting module 602, an encoding module 603, a conversion module 604, an embedding module 605, and an output module 606.
[0110] Among them, the determination module 601 is used to determine the surgical video and the original question related to the surgical video; the splitting module 602 is used to split the surgical video to obtain multiple video segments with a fixed number of frames; the encoding module 603 is used to encode each of the multiple video segments through a pre-trained video encoder to obtain abstract features; the conversion module 604 is used to convert the spatial dimension of the abstract feature to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter to obtain the processed abstract feature; the embedding module 605 is used to cross-embed the processed abstract feature with the text-assisted description to obtain a mixed abstract feature; the output module 606 is used to input the mixed abstract feature and the original question into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model.
[0111] Specifically, the surgical video processing device based on a multimodal large model provided by the present invention can implement all the method steps implemented in the above-mentioned surgical video processing method embodiment based on a multimodal large model, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.
[0112] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a surgical video processing method based on a multimodal large model, the method comprising: determining a surgical video and an original question related to the surgical video; splitting the surgical video to obtain a plurality of video segments with a fixed number of frames; encoding each of the plurality of video segments through a pre-trained video encoder to obtain an abstract feature; converting the spatial dimension of the abstract feature to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter to obtain a processed abstract feature; cross-embedding the processed abstract feature with the text auxiliary description to obtain a mixed abstract feature; inputting the mixed abstract feature and the original question into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model.
[0113] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surgical video processing method based on a multimodal large model provided by the above-mentioned methods, and the method includes: determining a surgical video and an original question related to the surgical video; splitting the surgical video to obtain multiple video segments with a fixed number of frames; encoding each of the multiple video segments through a pre-trained video encoder to obtain an abstract feature; converting the spatial dimension of the abstract feature to be consistent with the spatial dimension of a preset multimodal large model through a preset multimodal converter to obtain a processed abstract feature; cross-embedding the processed abstract feature with a text-assisted description to obtain a mixed abstract feature; inputting the mixed abstract feature and the original question into a preset multimodal large model to obtain a text answer content output by the preset multimodal large model.
[0115] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the surgical video processing method based on a multimodal large model provided by the above-mentioned methods, the method comprising: determining a surgical video and an original question related to the surgical video; splitting the surgical video to obtain a plurality of video segments with a fixed number of frames; encoding each of the plurality of video segments using a pre-trained video encoder to obtain an abstract feature; converting the spatial dimension of the abstract feature to be consistent with the spatial dimension of a preset multimodal large model using a preset multimodal converter to obtain a processed abstract feature; cross-embedding the processed abstract feature with a text-assisted description to obtain a mixed abstract feature; inputting the mixed abstract feature and the original question into a preset multimodal large model to obtain a text answer content output by the preset multimodal large model.
[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A surgical video processing method based on a multimodal large model, characterized in that: include: identifying the surgical video and the original question related to said surgical video; Splitting the surgical video to obtain multiple video segments with a fixed number of frames; Encoding each of the multiple video segments using a pre-trained video encoder to obtain an abstract feature; The spatial dimension of the abstract feature is converted to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter to obtain a processed abstract feature; Cross-embedding the processed abstract features with the text-assisted description to obtain mixed abstract features; The mixed abstract features and the original question are input into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model.
2. The surgical video processing method based on a multimodal large model according to claim 1, characterized in that: Before determining the surgical video and the original question related to the surgical video, the method further includes: A preset video encoder is trained to obtain a pre-trained video encoder, wherein the pre-trained video encoder is used to align surgical instrument video features with surgical instrument text features.
3. The surgical video processing method based on a multimodal large model according to claim 2 is characterized in that: The step of training a preset video encoder to obtain a pre-trained video encoder includes: Acquire a surgical video training sample, wherein the surgical video training sample includes at least one surgical instrument mark and a surgical instrument text feature corresponding to the surgical instrument mark; Performing instrument masking on the surgical video training sample by using a preset video encoder to obtain a masked video training sample; Encoding the masked video training sample by an encoder of the preset video encoder to obtain an encoded video training sample; Masking and reshaping the encoded video training sample by a decoder of the preset video encoder to obtain surgical instrument video features; Based on a preset loss function between the surgical instrument video features and the surgical instrument text features, the preset video encoder is updated to obtain a pre-trained video encoder.
4. The surgical video processing method based on a multimodal large model according to claim 1, characterized in that: The surgical video is split to obtain multiple video segments with a fixed number of frames, including: Converting the surgical video into continuous image frames; According to a preset target frame number, the continuous image frames are converted into a plurality of video segments, wherein the frame number of the video segments is the preset target frame number.
5. The surgical video processing method based on a multimodal large model according to claim 1, characterized in that: The cross-embedding of the processed abstract features and the text-assisted description to obtain mixed abstract features includes: Determine a timeline segment of the video segment corresponding to the processed abstract feature; Generate a text auxiliary description based on the timeline paragraph according to a preset text description template; The text auxiliary description is converted by a preset text token processor to obtain text abstract features; The processed abstract features are cross-embedded with the text abstract features to obtain mixed abstract features.
6. The surgical video processing method based on a multimodal large model according to claim 1, characterized in that: The step of inputting the mixed abstract features and the original question into the preset multimodal macro model to obtain text answer content output by the preset multimodal macro model includes: determining a problem type of the original problem; Calling the preset multimodal large model to determine a target pre-trained low-rank adaptation module based on the problem type; Based on the target pre-trained low-rank adaptation module, a text answer content is output according to the mixed abstract features and the original question.
7. A surgical video processing device based on a multimodal large model, characterized in that: include: A determination module, used for determining a surgical video and an original question related to the surgical video; A splitting module, used to split the surgical video to obtain multiple video segments with a fixed number of frames; An encoding module, configured to encode each of the plurality of video segments using a pre-trained video encoder to obtain an abstract feature; A conversion module, used to convert the spatial dimension of the abstract feature to be consistent with the spatial dimension of the preset multimodal large model through a preset multimodal converter, so as to obtain a processed abstract feature; An embedding module, used for cross-embedding based on the processed abstract features and text-assisted descriptions to obtain mixed abstract features; An output module is used to input the mixed abstract features and the original question into the preset multimodal large model to obtain the text answer content output by the preset multimodal large model.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the surgical video processing method based on the multimodal large model is implemented as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the surgical video processing method based on a multimodal large model is implemented as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the surgical video processing method based on a multimodal large model is implemented as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Video processing method and device, electronic equipment and storage medium
CN113923493A
Video question answering method and system based on cross-modal prompt learning
CN114996513A
Arbitrary focus segmentation and tracking system based on medical ultrasonic image
CN117350979A
Video question and answer method and electronic equipment
CN117917696A
Operation video description method and device, electronic equipment and storage medium
CN117994691A
