Surgical navigation method and device based on multi-modal large language model

Abstract

The application provides a surgery navigation method and device based on a multimodal large language model, wherein the method comprises: acquiring user voice input; resampling the user voice input to obtain resampled audio; performing mel-frequency spectrum conversion and normalization on the resampled audio to obtain voice features; determining a voice intent corresponding to the user voice input based on the voice features; performing similarity retrieval in a preset material document based on the voice intent by a retriever to obtain target application program interface text; splicing the target application program interface text and the voice intent to obtain text input; and inputting the text input and visual input into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine to perform surgery navigation. The application can provide more accurate real-time navigation in neurosurgery.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a surgical navigation method and apparatus based on a multimodal large language model. Background Technology

[0002] Precise navigation of deep brain structures is particularly important during neurosurgery. Although surgical navigation technology has made some progress, it still faces many challenges.

[0003] First, traditional surgery heavily relies on the skills of experienced neurosurgeons, and this dependence on individual expertise limits the universality and consistency of the procedure. Second, these methods still need improvement in terms of precision, especially in the complex intracranial environment where ensuring precise implantation positioning and spatial awareness is crucial. Furthermore, current procedures require surgeons to master complex robotic operating procedures that cannot be controlled through direct verbal commands, increasing both the difficulty of learning and operation and the workload of physicians.

[0004] This demonstrates that surgical navigation methods in related technologies present significant technical challenges in operation. Summary of the Invention

[0005] This invention provides a surgical navigation method and device based on a multimodal large language model, which solves the problem that existing surgical navigation methods are difficult to operate and enables real-time navigation in neurosurgery.

[0006] This invention provides a surgical navigation method based on a multimodal large language model, comprising the following steps: acquiring user voice input; resampling the user voice input to obtain resampled audio; performing Mel-spectrum transformation and normalization on the resampled audio to obtain speech features; determining the speech intent corresponding to the user voice input based on the speech features; performing a similarity search in preset documents based on the speech intent using a retrieval device to obtain target application programming interface (API) text, wherein the documents include API documentation and API manual; concatenating the target API text with the speech intent to obtain text input; inputting the text input and visual input together into a pre-trained large language model to obtain a text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation.

[0007] According to a surgical navigation method based on a multimodal large language model provided by the present invention, the step of performing Mel spectrum conversion and normalization on the resampled audio to obtain speech features includes: performing Mel spectrum conversion based on the user's speech input and the resampled audio to obtain a Mel spectrum; determining the average value of the Mel spectrum on the training dataset; and normalizing based on the Mel spectrum and the average value to obtain normalized speech features.

[0008] According to a surgical navigation method based on a multimodal large language model provided by the present invention, the step of determining the voice intent corresponding to the user's voice input based on the voice features includes: inputting the voice features into a voice encoder to obtain the voice coding features output by the voice encoder; inputting the voice coding features and a preset instruction set into an alignment network to obtain the voice intent output by the alignment network, wherein the voice intent corresponds to the user's voice input.

[0009] According to a surgical navigation method based on a multimodal large language model provided by the present invention, before inputting the text input and visual input together into a pre-trained large language model to obtain the text response output by the pre-trained large language model, the method further includes: acquiring a real-time video stream; inputting the real-time video stream into a visual encoder to obtain video features output by the visual encoder; and inputting the video features into a projection layer to obtain the visual input output by the projection layer.

[0010] According to the surgical navigation method based on a multimodal large language model provided by the present invention, before inputting the text input and visual input together into a pre-trained large language model to obtain the text response output by the pre-trained large language model, the method further includes: training a preset large language model based on LongLoRA to obtain a pre-trained large language model.

[0011] According to a surgical navigation method based on a multimodal large language model provided by the present invention, the step of inputting the text input and visual input together into a pre-trained large language model to obtain a text response output by the pre-trained large language model includes: generating a thought chain based on the text input and visual input using the pre-trained large language model; outputting a target operation application interface and target operation parameters based on the thought chain using the pre-trained large language model; and generating a text response based on the target operation application interface and the target operation parameters using the pre-trained large language model.

[0012] This invention also provides a surgical navigation device based on a multimodal large language model, comprising the following modules: an acquisition module for acquiring user voice input; a resampling module for resampling the user voice input to obtain resampled audio; a conversion module for performing Mel-spectrum conversion and normalization on the resampled audio to obtain speech features; a determination module for determining the speech intent corresponding to the user voice input based on the speech features; a retrieval module for performing a similarity search on preset documents based on the speech intent using a retrieval device to obtain target application programming interface (API) text, wherein the documents include API documentation and API manual; a concatenation module for concatenating the target API text with the speech intent to obtain text input; and an output module for inputting the text input and visual input together into a pre-trained large language model to obtain a text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the surgical navigation method based on a multimodal large language model as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the surgical navigation method based on a multimodal large language model as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the surgical navigation method based on a multimodal large language model as described above.

[0016] The surgical navigation method and apparatus based on a multimodal large language model provided by this invention resamples the user's voice input to obtain resampled audio. This allows for the acquisition of the user's voice input and its conversion into processable voice features. Mel-spectral conversion and normalization are performed on the resampled audio to extract useful voice features from the audio signal, supporting subsequent processing tasks. Based on the voice features, the voice intent corresponding to the user's voice input is determined, thereby accurately understanding the user's intent from the voice input. Similarity retrieval is performed in preset documents based on the voice intent, allowing direct retrieval of relevant application programming interface (API) documents and manuals to obtain target API text matching the voice intent. The target API text is concatenated with the voice intent to obtain text input. The text input and visual input are jointly input into a pre-trained large language model to obtain the text response output by the pre-trained large language model. Thus, by combining text modal input and visual modal input, the pre-trained large language model can more comprehensively understand the user's intent and the context of the technical documents, thereby outputting a text response for controlling a preset machine for surgical navigation. This solves the problem of high operational difficulty in existing surgical navigation methods. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the surgical navigation method based on a multimodal large language model provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the surgical navigation algorithm based on a multimodal large language model provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the surgical navigation algorithm based on a multimodal large language model provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the surgical navigation device based on a multimodal large language model provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Precise navigation of deep brain structures is crucial during neurosurgery. These surgeries are performed in complex neural environments near the brainstem, where even minor errors can lead to serious complications. However, current navigation technologies, despite some progress, face several challenges. First, traditional surgery heavily relies on the skills of experienced neurosurgeons, limiting the generalizability and consistency of the procedures. Second, these methods still need improvement in precision, especially in the complex intracranial environment where ensuring accurate implant placement and spatial awareness is paramount. Furthermore, current procedures require surgeons to master complex robotic operating procedures that cannot be controlled via direct verbal commands, increasing both the learning and operational difficulties and the surgeon's workload.

[0025] To address these issues, this invention proposes a surgical navigation method based on a multimodal large language model. This method is designed to improve surgical safety and efficiency by integrating multiple data inputs—including detailed preoperative MRI scans, real-time sensor feedback, direct text instructions from the surgical team, and intraoperative visual data streams. Through this integration, navigation can analyze and fuse visual and linguistic information in real time, enabling precise navigation and dynamic adjustments in complex surgical environments.

[0026] Specifically, the multimodal large language model of this invention utilizes advanced visual encoder and text decoder technologies to optimize the representation of visual and textual data from the ground up, achieving deep integration of cross-modal knowledge. By combining the visual modality mask autoencoder task with the textual modality next-to-toe prediction task, this system can not only comprehensively understand and integrate medical images and direct commands, but also effectively improve navigation accuracy and response speed during surgery by utilizing a joint generative pre-training strategy.

[0027] Overall, the surgical navigation method based on a multimodal large language model of the present invention significantly improves the accuracy and efficiency of surgery, has broad application potential, and provides a brand-new technical solution for future neurosurgical procedures.

[0028] The purpose of this invention is to improve the accuracy and efficiency of surgery by using a multimodal large language model-based intelligent navigation system for precise navigation of deep brain structures during surgery.

[0029] To achieve the above objectives, this invention utilizes a multimodal large language model for navigating deep brain microstructures near the brainstem and avoiding key brain tissues. The proposed multimodal large language model comprises three main network modules: the first module is a speech intent recognition module, which encodes and converts the doctor's speech as input into specific operational intents; the second module is a retrieval enhancement module, responsible for retrieving highly relevant pre-stored robot operation guidelines and application programming interface (API) guidelines, and inputting them as background information into the large model; the third module is a large language model, which accepts visual and textual features as input and ultimately generates the final text output and specific operation API, transmitting them to the robot to execute specific operations.

[0030] Optionally, the surgical navigation method based on a multimodal large language model in this embodiment can be executed by a server, by a terminal device (e.g., a surgical device), or by both a server and a terminal device. For example, the surgical navigation method based on a multimodal large language model in this embodiment can be executed by a terminal device.

[0031] Figure 1 This is a flowchart illustrating the surgical navigation method based on a multimodal large language model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps.

[0032] Step 101: Obtain user voice input.

[0033] In this embodiment of the invention, applied to a neurosurgical navigation scenario, obtaining user voice input can be achieved by receiving doctor's voice A in real time. () as voice input.

[0034] Step 102: Resample the user's voice input to obtain resampled audio.

[0035] In this embodiment of the invention, user voice input ( Resampling is performed to obtain the resampled audio. ).

[0036] Resampling user voice input typically involves converting the original audio signal's sampling rate to a different sampling rate. For example, reading the user's original voice audio from a source file or input stream (i.e., the user voice input mentioned above); determining a target sampling rate, the choice of which is based on application-specific requirements (such as the sampling rate required by the communication protocol); using a target resampling algorithm (e.g., linear interpolation, polynomial interpolation, window function interpolation, etc.) to convert the original audio signal from the original sampling rate to the target sampling rate (typically involving recalculation and interpolation for each sample point); and finally, saving the resampled audio data to a new file or outputting it to a subsequent audio processing pipeline.

[0037] Through the embodiments of the present invention, compatibility of input audio files with specific devices or encoding standards can be ensured.

[0038] Step 103: Perform Mel spectrum transformation and normalization on the resampled audio to obtain speech features.

[0039] In this embodiment of the invention, Mel spectrum conversion is performed based on user voice input and resampled audio to obtain Mel spectrum; the average value of Mel spectrum on the training dataset is determined; and normalization is performed based on Mel spectrum and average value to obtain normalized speech features.

[0040] Here, the resampled audio undergoes Mel-spectrogram transformation, converting the frequency axis of the audio signal to a Mel scale to better simulate the auditory characteristics of the human ear. Subsequently, the data is normalized to give it specific properties (such as a mean of 0 and a standard deviation of 1).

[0041] After Mel spectrum transformation and normalization, the resulting Mel spectrum features (speech features) can be used in subsequent machine learning or deep learning models.

[0042] Step 104: Based on speech features, determine the speech intent corresponding to the user's speech input.

[0043] In this embodiment of the invention, speech features are input to a speech encoder to obtain speech coding features output by the speech encoder; the speech coding features and a preset instruction set are input to an alignment network to obtain speech intent output by the alignment network, wherein the speech intent corresponds to the user's speech input.

[0044] Step 105: The retrieval device performs a similarity search on preset documents based on voice intent to obtain the target application interface text, wherein the documents include application interface documents and application interface manuals.

[0045] In this embodiment of the invention, based on the obtained voice intent Retrieve pre-stored Application Programming Interface (API) documentation and manuals using a search engine. Perform a similarity search to obtain the closest API and its description. For details, please refer to the following formula (1):

[0046] (1)

[0047] in, Indicates the target application interface text. The function is used to sequentially select a target number (of elements with the highest similarity) from a set or array. Indicates the search engine. This refers to the application programming interface (API) documentation and the API manual.

[0048] Here, the documentation may also include application documents or technical manuals for surgical equipment (e.g., surgical robots), which can be pre-set according to the actual application scenario.

[0049] Here, Application Programming Interface (API) documentation and API manuals are documents used to describe and define software application programming interfaces (APIs). They provide standardized interface descriptions and related information.

[0050] Step 106: Concatenate the target application interface text with the voice intent to obtain the text input.

[0051] In this embodiment of the invention, the obtained target application interface text ( ) and speech intent ( The set of text inputs for the large language model is obtained by obtaining the text input of the large language model. For details, please refer to the following formula (2):

[0052] (2)

[0053] in, The text input represents the large language model. This indicates a splicing operation. Indicates the target application interface text. Indicates the meaning of speech.

[0054] Step 107: Input both text input and visual input into a pre-trained large language model to obtain the text response output by the pre-trained large language model. The text response is used to control the preset machine for surgical navigation.

[0055] In this embodiment of the invention, the real-time video stream Input visual encoder The visual input, aligned with the large model, is obtained through a projection layer. Together with the previously obtained text input Input a large language model.

[0056] In this embodiment of the invention, the large language model itself is pre-trained with LongLoRA and has the ability to support long windows, which can accommodate long API guidelines and their descriptions. The large language model ultimately outputs text answers through the thought chain and can output API to control the machine for navigation.

[0057] refer to Figure 2 , Figure 2 This is a schematic diagram of the surgical navigation algorithm based on a multimodal large language model provided by the present invention. The navigation system includes: dynamic API human-computer interaction, long window input, and voice activation operation.

[0058] Here, users control the Large Vision Language Model (LVLM) via voice and receive multimodal feedback from the LVLM; the surgical equipment sends multimodal input to the LVLM and receives the output of the LVLM for navigation and operation of the machine.

[0059] Through the steps described in this embodiment of the invention, user voice input is acquired; the user voice input is resampled to obtain resampled audio; the resampled audio undergoes Mel-spectrum transformation and normalization to obtain speech features; based on the speech features, the speech intent corresponding to the user voice input is determined; a retrieval device performs a similarity search on preset documents based on the speech intent to obtain the target application interface text, wherein the documents include application interface documents and application interface manuals; the target application interface text and the speech intent are concatenated to obtain text input; the text input and visual input are jointly input into a pre-trained large language model to obtain the text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation. This effectively improves the accuracy and response speed of navigation during surgery and reduces the difficulty of application, thereby solving the problem of high operational difficulty in existing surgical navigation methods.

[0060] According to the surgical navigation method based on a multimodal large language model provided by the present invention, resampled audio undergoes Mel-spectrum transformation and normalization to obtain speech features, including:

[0061] Mel spectrum is obtained by performing Mel spectrum conversion based on user voice input and resampled audio;

[0062] Determine the average value of the Mel spectrum on the training dataset;

[0063] Normalized speech features are obtained by normalizing the Mel spectrum and its mean value.

[0064] Resample audio ( The signal is converted to a Mel spectrum, normalized, and then input into the speech encoder for instruction alignment. The specific operations are as follows:

[0065] First, perform the Mel spectrum conversion operation, which can be referred to in the following formula (3):

[0066] (3)

[0067] in, Represents the Mel spectrum. This indicates a Mel-spectrum conversion operation. This indicates user voice input. Indicates resampled audio. Indicates the number of channels. Indicates window size. Indicates step frequency.

[0068] Secondly, the obtained The Mel spectrum is normalized, and the specific steps can be found in the following formula (4):

[0069] (4)

[0070] in, This represents the normalized speech features. Represents the Mel spectrum. This represents the average value of the Mel spectrum on the training dataset.

[0071] Through the embodiments of the present invention, Mel spectrum conversion transforms the spectral representation of audio signals to the Mel frequency scale, which is closer to the auditory perception characteristics of the human ear; normalization can eliminate the influence of differences in dimensions between different audio signals, making subsequent processing more fair and accurate.

[0072] According to the present invention, a surgical navigation method based on a multimodal large language model determines the speech intent corresponding to the user's speech input based on speech features, including:

[0073] The speech features are input into the speech encoder to obtain the speech coding features output by the speech encoder;

[0074] The speech coding features and a preset instruction set are input into the alignment network to obtain the speech intent output by the alignment network, where the speech intent corresponds to the user's speech input.

[0075] In this embodiment of the invention, the obtained normalized speech features are input into the speech encoder, and the extracted speech coding features are aligned with the pre-stored instruction set. Specifically, the following formula (5) can be referred to:

[0076] (5)

[0077] in, Indicates the meaning of speech. Indicates the alignment network. Indicates a voice encoder. This represents the normalized speech features.

[0078] Here, the alignment network first extracts meaningful features from the speech signal, such as Mel-frequency cepstral coefficients (MFCCs), which reflect the spectral and temporal dynamics of speech. Then, it is trained using labeled speech and text data. During training, the alignment network learns how to align speech features with their corresponding text content. In the inference phase, the alignment network accepts new speech input and outputs the alignment result with the text content. This alignment result is typically represented by the temporal correspondence between text and speech features.

[0079] Through the embodiments of the present invention, in speech recognition, the alignment network can align the obtained speech features according to their intent, thereby achieving accurate parsing and understanding of speech signals.

[0080] According to the surgical navigation method based on a multimodal large language model provided by the present invention, before inputting both text input and visual input into a pre-trained large language model to obtain the text response output by the pre-trained large language model, the method further includes:

[0081] Acquire real-time video stream;

[0082] The real-time video stream is input into the visual encoder to obtain the video features output by the visual encoder;

[0083] The video features are input into the projection layer to obtain the visual input output by the projection layer.

[0084] In this embodiment of the invention, a real-time video stream is input into a visual encoder, and a visual input aligned with a large model is obtained through a projection layer. This visual input is then input into a large language model along with the previously obtained text input.

[0085] In this embodiment of the invention, the visual encoder is typically a deep learning model, such as a convolutional neural network (CNN), which can extract target features from the input video frames. These target features include edges, textures, colors, shapes, etc., or higher-level semantic information (such as object categories, actions, etc.).

[0086] Each frame of the real-time video stream is input into the visual encoder, which outputs a corresponding (video) feature vector or feature map. Video features represent an abstract representation of the video content and are used for subsequent processing or analysis.

[0087] In this embodiment of the invention, the projection layer can be a transformation layer, such as a fully connected layer, a convolutional layer, or a pooling layer, or it can be a more complex network structure, such as a recurrent neural network (RNN) or a self-attention mechanism. The projection layer is used to convert the visual features extracted from the visual encoder into the desired visual input format (i.e., video input).

[0088] After the video features are input into the projection layer, they undergo a series of transformations to obtain the final visual input. This visual input can be data used for further analysis (such as classification, detection, tracking, etc.) or data used to generate visual effects (such as augmented reality, virtual reality, etc.).

[0089] According to the surgical navigation method based on a multimodal large language model provided by the present invention, before inputting the text input and visual input together into a pre-trained large language model to obtain the text response output by the pre-trained large language model, the method further includes:

[0090] The pre-trained large language model is obtained by training a pre-defined large language model based on LongLoRA.

[0091] In this embodiment of the invention, the preset large language model itself is pre-trained with LongLoRA and has the ability to support long windows, which can accommodate long API guidelines and their descriptions. The large language model outputs a text answer through the thought chain and can output API to control the machine for navigation.

[0092] Here, LongLoRA is a pre-training technique optimized for long text or long sequence data. Traditional language models may encounter problems such as loss of contextual information or limited computational resources when processing long text. LongLoRA improves the model structure or training strategy, enabling the model to better capture information in long text; the ability to support long windows means that the model can handle long text inputs containing a large amount of information, such as complete API guidelines and their detailed descriptions.

[0093] In this embodiment of the invention, the entire multimodal large-scale language model is optimized in conjunction with the above embodiments. Specifically, training is divided into two stages. The first stage is visual pre-training, aligning the visual and textual aspects. The second stage is unified instruction fine-tuning training, which uses a real dataset to train the model so that it can call APIs to control the machine for navigation.

[0094] According to the present invention, a surgical navigation method based on a multimodal large language model is provided, in which text input and visual input are jointly input into a pre-trained large language model to obtain a text response output by the pre-trained large language model, including:

[0095] A thought chain is generated based on text and visual input using a pre-trained large language model.

[0096] Based on the thought chain, a pre-trained large language model outputs the target operation application interface and target operation parameters.

[0097] A pre-trained large language model generates text responses based on the target operation application interface and target operation parameters.

[0098] In this embodiment of the invention, the large language model receives input from different modalities, including text modalities (including target application interface text and voice intent) and visual modalities (including video data obtained from real-time video streams). These inputs together constitute the context that the large language model needs to understand and process.

[0099] Here, the large language model uses the knowledge and reasoning ability gained in pre-training to integrate the input textual and visual information to form a series of internal reasoning steps or thought chains. These thought chains are used to parse, understand, associate, and reason about the input information in order to build a comprehensive understanding of the input content.

[0100] After the large language model understands the input content, it determines the target operation application programming interface (API) to be called and generates the target operation parameters based on the input information and internal reasoning process (thinking chain).

[0101] For example, based on understanding the API function and matching the input information with the API function, the API with the highest matching degree and the target operation parameters are obtained to execute the required task. After generating the target operation API and target operation parameters, the large language model can call the API through internal mechanisms or external interfaces. After the API call is completed, the large language model needs to parse the returned result. Finally, the large language model will generate an easy-to-understand text answer based on the parsed result and input information. The text answer includes the result of the API call, the explanation of the result, and any relevant suggestions or explanations.

[0102] refer to Figure 3 , Figure 3 This is a schematic diagram of the surgical navigation algorithm based on a multimodal large language model provided by the present invention.

[0103] Here, the doctor inputs a voice command, the voice activation module receives the voice command and converts it into a speech memspectral graph, which is then input to the voice encoder. The obtained speech features are aligned with a preset command set to obtain the voice intent. The voice intent is then input to the dynamic API information retrieval enhancement module, which searches the preset API documents / guides based on the voice intent to obtain the number of closest target application interface documents (Top-K APIs). The command input is fused with the target application interface documents and target application interface text (API-Former) to obtain a text token (text input). The real-time video stream from the multimodal input is input to the visual encoder, and after passing through the projection layer, a visual token (visual input) is obtained. The text token and visual token are input to the pre-trained large language model in the long window input module to obtain the API combination output of the large language model. The API combination output is used to control the navigation and operation of the preset machine.

[0104] The long window input module includes a large visual language model (LVLM) and dynamic execution. The large visual language model includes a large language model (LLM), which includes an encoding process (Embedding), normalization, a multi-head self-attention mechanism, normalization, and feedforward. Here, the input of the multi-head self-attention mechanism includes navigation instructions (auxiliary descriptions, extraction APIs, and extraction guidelines).

[0105] The following describes an example of the surgical navigation method based on a multimodal large language model provided by this invention in a practical application scenario.

[0106] The core of this invention lies in solving the problem of precise navigation of deep brain structures during surgery in the field of neurosurgery. Given the limitations of existing navigation technologies, including over-reliance on experienced neurosurgeons and insufficient accuracy and operational consistency, this invention proposes an intelligent navigation system based on a multimodal large language model (LVLM). This system integrates preoperative MRI scans, real-time sensor feedback, text commands from the surgical team, and intraoperative visual data streams. Through efficient visual encoders and text decoders, it optimizes the representation of visual and text data from the ground up, achieving deep fusion of cross-modal knowledge.

[0107] Specifically, by combining a visual modality mask autoencoder task with a text modality next-to-toe prediction task, this system can not only comprehensively understand and integrate medical images and direct commands, but also effectively improve navigation accuracy and response speed during surgery through a joint generative pre-training strategy. Overall, the intelligent navigation system of this invention significantly improves surgical precision and application efficiency, has broad application potential, and provides a novel technical solution for future neurosurgical procedures.

[0108] This invention includes the following steps:

[0109] Step S0: Receive doctor's voice message As a voice input;

[0110] Step S1: Resample the audio to obtain resampled audio. ;

[0111] Step S2, resample the audio The signal is converted to Mel spectrum, normalized, and then input into the speech encoder for instruction alignment. The specific steps are as follows:

[0112] First, perform the Mel spectrum conversion operation, which can be referred to in the following formula (6):

[0113] (6)

[0114] in, Represents the Mel spectrum. This indicates a Mel-spectrum conversion operation. This indicates user voice input. Indicates resampled audio. Indicates the number of channels. Indicates window size. Indicates step frequency.

[0115] Secondly, the obtained The Mel spectrum is normalized, and the specific steps can be found in the following formula (7):

[0116] (7)

[0117] in, This represents the normalized speech features. Represents the Mel spectrum. This represents the average value of the Mel spectrum on the training dataset.

[0118] Finally, the normalized speech features are input into the speech encoder, and the extracted speech coding features are aligned with the pre-stored instruction set. For details, please refer to the following formula (8):

[0119] (8)

[0120] in, Indicates the meaning of speech. Indicates the alignment network. Indicates a voice encoder. This represents the normalized speech features.

[0121] Step S3: Based on the obtained intent C, perform a similarity search on the pre-stored API documentation and manual using the searcher R to obtain the closest API and its description T. API For details, please refer to the following formula (9):

[0122] (9)

[0123] in, Indicates the target application interface text. The function is used to sequentially select a target number (of elements with the highest similarity) from a set or array. Indicates the search engine. This refers to the application programming interface (API) documentation and the API manual.

[0124] Step S4, obtain the target application interface text ( ) and speech intent ( The set of text inputs for the large language model is obtained by obtaining the text input of the large language model. For details, please refer to the following formula (10):

[0125] (10)

[0126] in, The text input represents the large language model. This indicates a splicing operation. Indicates the target application interface text. Indicates the meaning of speech.

[0127] Step S5: Input the real-time video stream V into the visual encoder V. E The visual input V, aligned with the large model, is obtained through a projection layer. input Together with the previously obtained text input T input Input a large language model.

[0128] In step S6, the large language model itself is pre-trained with LongLoRA and has the ability to support long windows, which can accommodate long API guidelines and their descriptions. The large language model outputs a text answer through the thought chain and can output the API to control the machine for navigation.

[0129] The surgical navigation device based on a multimodal large language model provided by the present invention will be described below. The surgical navigation device based on a multimodal large language model described below and the surgical navigation method based on a multimodal large language model described above can be referred to and correspond to each other.

[0130] refer to Figure 4 , Figure 4 This is a schematic diagram of the surgical navigation device based on a multimodal large language model provided by the present invention, which includes an acquisition module 401, a resampling module 402, a conversion module 403, a determination module 404, a retrieval module 405, a splicing module 406, and an output module 407.

[0131] Module 401 is used to acquire user voice input;

[0132] The resampling module 402 is used to resample the user's voice input to obtain resampled audio;

[0133] The conversion module 403 is used to perform Mel spectrum conversion and normalization on the resampled audio to obtain speech features;

[0134] The determination module 404 is used to determine the voice intent corresponding to the user's voice input based on voice features;

[0135] The retrieval module 405 is used to perform similarity retrieval in preset documents based on voice intent through a retrieval device to obtain the target application interface text, wherein the documents include application interface documents and application interface manuals.

[0136] The splicing module 406 is used to splice the target application interface text and voice intent to obtain text input;

[0137] The output module 407 is used to input both text input and visual input into a pre-trained large language model to obtain a text response output by the pre-trained large language model. The text response is used to control a preset machine for surgical navigation.

[0138] Specifically, the surgical navigation device based on a multimodal large language model provided by the present invention can realize all the method steps implemented in the above-mentioned surgical navigation method embodiment based on a multimodal large language model, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0139] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, as shown below. Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a surgical navigation method based on a multimodal large language model. This method includes: acquiring user voice input; resampling the user voice input to obtain resampled audio; performing Mel-spectrum transformation and normalization on the resampled audio to obtain speech features; determining the speech intent corresponding to the user voice input based on the speech features; performing a similarity search on preset documents based on the speech intent using a retrieval device to obtain target application interface text, wherein the documents include application interface documentation and application interface manual; concatenating the target application interface text with the speech intent to obtain text input; and inputting the text input and visual input together into a pre-trained large language model to obtain a text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation.

[0140] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surgical navigation method based on a multimodal large language model provided by the above methods. The method includes: acquiring user voice input; resampling the user voice input to obtain resampled audio; performing Mel-spectrum transformation and normalization on the resampled audio to obtain speech features; determining the speech intent corresponding to the user voice input based on the speech features; performing a similarity search in a preset document based on the speech intent using a retrieval device to obtain target application interface text, wherein the document includes application interface documentation and application interface manual; concatenating the target application interface text with the speech intent to obtain text input; and inputting the text input and visual input together into a pre-trained large language model to obtain a text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation.

[0142] On another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the surgical navigation method based on a multimodal large language model provided by the above methods. This method includes: acquiring user voice input; resampling the user voice input to obtain resampled audio; performing Mel-spectral conversion and normalization on the resampled audio to obtain speech features; determining the speech intent corresponding to the user voice input based on the speech features; performing a similarity search on preset documents based on the speech intent using a retrieval device to obtain target application interface text, wherein the documents include application interface documents and application interface manuals; concatenating the target application interface text with the speech intent to obtain text input; and inputting the text input and visual input together into a pre-trained large language model to obtain a text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surgical navigation method based on a multimodal large language model, characterized in that, include: Obtain user voice input; The user's voice input is resampled to obtain resampled audio; The resampled audio is subjected to Mel-spectrum transformation and normalization to obtain speech features; Based on the voice features, determine the voice intent corresponding to the user's voice input; The target application interface text is obtained by performing a similarity search on preset documents based on the voice intent using a search engine. The documents include application interface documents and application interface manuals. Text input is obtained by concatenating the target application interface text with the voice intent. The text input and visual input are fed together into a pre-trained large language model to obtain a text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation; The process of performing Mel-spectral transformation and normalization on the resampled audio to obtain speech features includes: Based on the user's voice input and the resampled audio, a Mel spectrum conversion is performed to obtain the Mel spectrum: ； in, This refers to the Mel spectrum. This indicates a Mel-spectrum conversion operation. This indicates the user's voice input. This refers to the resampled audio. Indicates the number of channels. Indicates window size. Indicates step frequency; Determine the average value of the Mel spectrum on the training dataset; Based on the Mel spectrum and the average value, normalization is performed to obtain the normalized speech features: ； in, This represents the normalized speech features. This refers to the Mel spectrum. This represents the average value of the Mel spectrum on the training dataset.

2. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that, Determining the voice intent corresponding to the user's voice input based on the voice features includes: The speech features are input into the speech encoder to obtain the speech coding features output by the speech encoder; The speech coding features and a preset instruction set are input into an alignment network to obtain the speech intent output by the alignment network, wherein the speech intent corresponds to the user's speech input.

3. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that, Before inputting the text input and visual input together into a pre-trained large language model to obtain the text response output by the pre-trained large language model, the method further includes: Acquire real-time video stream; The real-time video stream is input to the visual encoder to obtain the video features output by the visual encoder; The video features are input into the projection layer to obtain the visual input output by the projection layer.

4. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that, Before inputting the text input and visual input together into a pre-trained large language model to obtain the text response output by the pre-trained large language model, the method further includes: The pre-trained large language model is obtained by training a pre-defined large language model based on LongLoRA.

5. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that, The step of inputting the text input and visual input together into a pre-trained large language model to obtain the text response output by the pre-trained large language model includes: Based on the text and visual inputs, a thought chain is generated using a pre-trained large language model. Based on the thought chain, the pre-trained large language model outputs the target operation application interface and the target operation parameters. The pre-trained large language model generates a text response based on the target operation application interface and the target operation parameters.

6. A surgical navigation device based on a multimodal large language model, characterized in that, include: The acquisition module is used to acquire user voice input; The resampling module is used to resample the user's voice input to obtain resampled audio; The conversion module is used to perform Mel-spectrum conversion and normalization on the resampled audio to obtain speech features; The determination module is used to determine the voice intent corresponding to the user's voice input based on the voice features; The retrieval module is used to perform a similarity search in a preset document based on the voice intent using a retrieval device to obtain the target application interface text, wherein the document includes an application interface document and an application interface manual. The splicing module is used to splice the target application interface text with the voice intent to obtain text input; The output module is used to input the text input and visual input together into a pre-trained large language model to obtain the text response output by the pre-trained large language model, wherein the text response is used to control a preset machine for surgical navigation; The process of performing Mel-spectral transformation and normalization on the resampled audio to obtain speech features includes: Based on the user's voice input and the resampled audio, a Mel spectrum conversion is performed to obtain the Mel spectrum: ； in, This refers to the Mel spectrum. This indicates a Mel-spectrum conversion operation. This indicates the user's voice input. This refers to the resampled audio. Indicates the number of channels. Indicates window size. Indicates step frequency; Determine the average value of the Mel spectrum on the training dataset; Based on the Mel spectrum and the average value, normalization is performed to obtain the normalized speech features: ； in, This represents the normalized speech features. This refers to the Mel spectrum. This represents the average value of the Mel spectrum on the training dataset.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the surgical navigation method based on a multimodal large language model as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the surgical navigation method based on a multimodal large language model as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the surgical navigation method based on a multimodal large language model as described in any one of claims 1 to 5.

Images