Surgical navigation method and device based on multi-modal large language model

Abstract

The invention provides a surgical navigation method and device based on a multi-mode large language model, and the method comprises the steps: obtaining the voice input of a user; re-sampling the user voice input to obtain a re-sampled audio; performing Mel spectrum conversion and normalization on the resampled audio to obtain voice features; determining a voice intention corresponding to the voice input of the user based on the voice features; performing similarity retrieval in a preset data document based on the voice intention through a retriever to obtain a target application program interface text; splicing the target application program interface text and the voice intention to obtain text input; the text input and the visual input are jointly input into a pre-trained large language model, a text answer output by the pre-trained large language model is obtained, and the text answer is used for controlling a preset machine to conduct surgical navigation. According to the invention, more accurate real-time navigation can be provided in the neurosurgery operation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a surgical navigation method and device based on a multimodal large language model. Background Art

[0002] When performing neurosurgery, accurate navigation of deep brain structures is particularly important. Although certain progress has been made in related technologies, surgical navigation technology still faces many challenges.

[0003] First, traditional surgery relies heavily on the operational skills of experienced neurosurgeons. This reliance on personal skills limits the universality and consistency of surgery. Second, these methods still need to be improved in terms of accuracy, especially in the complex brain environment, where it is crucial to ensure accurate positioning and spatial perception of implants. In addition, existing operating methods require surgeons to master complex robotic operating procedures and cannot be controlled by direct verbal instructions, which not only increases the difficulty of learning and operation, but also increases the workload of doctors.

[0004] It can be seen that the surgical navigation method in the related art has the technical problem of high operating difficulty. Summary of the invention

[0005] The present invention provides a surgical navigation method and device based on a multimodal large language model, which are used to solve the defect of high operating difficulty in the surgical navigation method in the prior art and provide real-time navigation in neurosurgery.

[0006] The present invention provides a surgical navigation method based on a multimodal large language model, comprising the following steps: obtaining user voice input; resampling the user voice input to obtain resampled audio; performing Mel spectrum conversion and normalization on the resampled audio to obtain voice features; determining the voice intent corresponding to the user voice input based on the voice features; performing similarity retrieval in a preset data document based on the voice intent through a retriever to obtain a target application interface text, wherein the data document includes an application interface document and an application interface manual; splicing the target application interface text with the voice intent to obtain a text input; inputting the text input and the visual input together into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine for surgical navigation.

[0007] According to a surgical navigation method based on a multimodal large language model provided by the present invention, the resampled audio is subjected to Mel spectrum conversion and normalization to obtain speech features, including: performing Mel spectrum conversion based on the user voice input and the resampled audio to obtain a Mel spectrum; determining an average value of the Mel spectrum on a training data set; and normalizing based on the Mel spectrum and the average value to obtain normalized speech features.

[0008] According to a surgical navigation method based on a multimodal large language model provided by the present invention, the speech intention corresponding to the user voice input is determined based on the speech features, including: inputting the speech features into a speech encoder to obtain speech coding features output by the speech encoder; inputting the speech coding features and a preset instruction set into an alignment network to obtain the speech intention output by the alignment network, wherein the speech intention corresponds to the user voice input.

[0009] According to a surgical navigation method based on a multimodal large language model provided by the present invention, before the text input and the visual input are inputted together into a pre-trained large language model to obtain a text answer outputted by the pre-trained large language model, the method further includes: acquiring a real-time video stream; inputting the real-time video stream into a visual encoder to obtain video features outputted by the visual encoder; and inputting the video features into a projection layer to obtain a visual input outputted by the projection layer.

[0010] According to a surgical navigation method based on a multimodal large language model provided by the present invention, before the text input and the visual input are jointly input into the pre-trained large language model to obtain the text answer output by the pre-trained large language model, the method also includes: training the preset large language model based on LongLoRA to obtain the pre-trained large language model.

[0011] According to a surgical navigation method based on a multimodal large language model provided by the present invention, the text input and visual input are jointly input into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, including: generating a thought chain based on the text input and visual input through the pre-trained large language model; outputting a target operation application program interface and target operation parameters based on the thought chain through the pre-trained large language model; and generating a text answer based on the target operation application program interface and the target operation parameters through the pre-trained large language model.

[0012] The present invention also provides a surgical navigation device based on a multimodal large language model, comprising the following modules: an acquisition module, used to acquire user voice input; a resampling module, used to resample the user voice input to obtain resampled audio; a conversion module, used to perform Mel spectrum conversion and normalization on the resampled audio to obtain voice features; a determination module, used to determine the voice intention corresponding to the user voice input based on the voice features; a retrieval module, used to perform similarity retrieval in a preset data document based on the voice intention through a retriever to obtain a target application interface text, wherein the data document includes an application interface document and an application interface manual; a splicing module, used to splice the target application interface text with the voice intention to obtain a text input; an output module, used to input the text input and the visual input together into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine for surgical navigation.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the surgical navigation method based on the multimodal large language model as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a surgical navigation method based on a multimodal large language model as described in any one of the above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described surgical navigation methods based on a multimodal large language model.

[0016] The surgical navigation method and device based on the multimodal large language model provided by the present invention can obtain the user's voice input by resampling the user's voice input to obtain the resampled audio, and can obtain the user's voice input and convert it into processable voice features; perform Mel spectrum conversion and normalization on the resampled audio, and can extract useful voice features from the audio signal to support subsequent processing tasks; determine the voice intention corresponding to the user's voice input based on the voice features, thereby accurately understanding the user's intention from the user's voice input; perform similarity retrieval in the preset data document based on the voice intention, and can directly retrieve the relevant application interface document and application interface manual through the voice intention to obtain the target application interface text that conforms to the voice intention; splice the target application interface text with the voice intention to obtain the text input; input the text input and the visual input into the pre-trained large language model together to obtain the text answer output by the pre-trained large language model, thereby combining the text modality input and the visual modality input, so that the pre-trained large language model can more comprehensively understand the user's intention and the technical document context, thereby outputting the text answer for controlling the preset machine for surgical navigation, thereby solving the defect of the surgical navigation method in the prior art that the operation is difficult. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the surgical navigation method based on the multimodal large language model provided by the present invention.

[0019] Figure 2 It is a structural schematic diagram of the surgical navigation algorithm based on the multimodal large language model provided by the present invention.

[0020] Figure 3 It is a model schematic diagram of the surgical navigation algorithm based on the multimodal large language model provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of a surgical navigation device based on a multimodal large language model provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] When performing surgery in the field of neurosurgery, accurate navigation of deep brain structures is particularly important. This type of surgery needs to be performed in a complex neural environment close to the brainstem, where any slight error may lead to serious complications. However, existing navigation technologies, despite certain progress, still face many challenges. First, traditional surgery relies heavily on the operational skills of experienced neurosurgeons, and this dependence on personal skills limits the universality and consistency of surgery. Secondly, these methods still need to be improved in terms of accuracy, especially in the complex intra-brain environment, where it is crucial to ensure accurate positioning and spatial perception of implants. In addition, existing operating methods require surgeons to master complex robotic operating procedures and cannot be controlled by direct verbal instructions, which not only increases the difficulty of learning and operation, but also increases the workload of doctors.

[0025] To address these issues, this paper proposes a surgical navigation method based on a multimodal large language model. The method is designed to improve the safety and efficiency of surgery by integrating multiple data inputs, including detailed preoperative MRI scans, real-time sensor feedback, direct text instructions from the surgical team, and intraoperative visual data streams. Through this integrated approach, navigation can parse and fuse visual and language information in real time, achieving precise navigation and dynamic adjustment of complex surgical environments.

[0026] Specifically, the multimodal large language model of the present invention uses advanced visual encoder and text decoder technology to optimize the representation of visual and text data from the bottom up, and achieve in-depth integration of cross-modal knowledge. By combining the masked autoencoder task of the visual modality with the next token prediction task of the text modality, the system can not only fully understand and integrate medical images and direct instructions, but also use a joint generative pre-training strategy to effectively improve navigation accuracy and response speed during surgery.

[0027] In general, the surgical navigation method based on the multimodal large language model of the present invention significantly improves the accuracy and application efficiency of surgery, has broad application potential, and provides a new technical solution for future neurosurgery.

[0028] The purpose of the present invention is to use an intelligent navigation system of a multimodal large language model to accurately navigate deep brain structures during surgery, thereby improving the accuracy and application efficiency of the surgery.

[0029] In order to achieve the above-mentioned purpose, the present invention utilizes a multimodal large language model for navigating deep brain microstructures close to the brainstem and avoiding key brain tissues. The multimodal large language model proposed in the present invention includes three main network modules. The first module is a speech intention recognition module, which is used to take the doctor's speech as input, encode and convert it into a specific operation intention; the second module is a retrieval enhancement module, which is responsible for retrieving the pre-stored robot operation guide and application interface guide with high relevance, and input them into the large model together as background information; the third module is a large language model, which is used to accept visual and text features as input and finally generate the final text output and the specific operation application interface and transmit it to the robot to perform specific operations.

[0030] Optionally, the surgical navigation method based on a multimodal large language model in the embodiment of the present application can be executed by a server, or by a terminal device (for example, a surgical device), or can be executed jointly by a server and a terminal device, taking the execution of the surgical navigation method based on a multimodal large language model in this embodiment by a terminal device as an example.

[0031] Figure 1 is a flow chart of a surgical navigation method based on a multimodal large language model provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0032] Step 101, obtaining user voice input.

[0033] In the embodiment of the present invention, when applied to the neurosurgery navigation scenario, obtaining the user voice input may be to receive the doctor's voice A in real time ( ) as voice input.

[0034] Step 102: resample the user's voice input to obtain resampled audio.

[0035] In the embodiment of the present invention, the user's voice input ( ) to resample and obtain the resampled audio ( ).

[0036] Resampling the user voice input usually involves converting the sampling rate of the original audio signal to another different sampling rate, for example, reading the user's original voice audio (i.e., the user voice input mentioned above) from a source file or input stream; determining a target sampling rate, the selection of which is based on the requirements of a specific application (such as the sampling rate required by a communication protocol); using a target resampling algorithm (e.g., linear interpolation, polynomial interpolation, window function interpolation, etc.) to convert the original audio signal from the original sampling rate to the target sampling rate (usually involving recalculation and interpolation of each sample point), and finally, saving the resampled audio data to a new file or outputting it to a subsequent audio processing pipeline.

[0037] Through the embodiments of the present invention, the compatibility of the input audio file with a specific device or encoding standard can be ensured.

[0038] Step 103: Perform Mel spectrum conversion and normalization on the resampled audio to obtain speech features.

[0039] In an embodiment of the present invention, a Mel spectrum conversion is performed based on the user voice input and the resampled audio to obtain a Mel spectrum; an average value of the Mel spectrum on the training data set is determined; and normalization is performed based on the Mel spectrum and the average value to obtain normalized speech features.

[0040] Here, the resampled audio is converted to a Mel-spectrogram to convert the frequency axis of the audio signal to a Mel scale to better simulate the auditory characteristics of the human ear. Subsequently, normalization adjusts the data to have specific properties (such as a mean of 0 and a standard deviation of 1).

[0041] After Mel spectrum conversion and normalization, the obtained Mel spectrum features (speech features) can be used for subsequent machine learning or deep learning models.

[0042] Step 104: Determine the voice intention corresponding to the user's voice input based on the voice features.

[0043] In an embodiment of the present invention, speech features are input into a speech encoder to obtain speech coding features output by the speech encoder; the speech coding features and a preset instruction set are input into an alignment network to obtain speech intentions output by the alignment network, wherein the speech intentions correspond to the user speech input.

[0044] Step 105, a searcher performs similarity search in a preset data document based on the speech intent to obtain a target application program interface text, wherein the data document includes an application program interface document and an application program interface manual; In the embodiment of the present invention, according to the obtained speech intention Go to the pre-stored API (Application Programming Interface) documents and manuals through the search engine Perform similarity search to obtain the closest API and its description required , please refer to the following formula (1): (1)

[0045] in, Represents the target application interface text, The function is used to sequentially select the target number of elements (with the greatest similarity) from a set or array. Represents a retriever, Indicates application program interface document and application program interface manual.

[0046] Here, the data documents may also include application documents or technical manuals of surgical equipment (eg, surgical robots), which may be pre-set according to actual application scenarios.

[0047] Here, the application interface document and application interface manual are documents used to describe and define the software application interface, which provides standardized interface descriptions and related information.

[0048] Step 106, concatenating the target application program interface text with the voice intent to obtain text input.

[0049] In the embodiment of the present invention, the target application program interface text ( ) and voice intent ( ) collection to obtain the text input of the large language model ( ), please refer to the following formula (2): (2)

[0050] in, represents the text input of the large language model, Represents a splicing operation, Represents the target application interface text, Indicates voice intent.

[0051] Step 107, input the text input and the visual input into the pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine for surgical navigation.

[0052] In the embodiment of the present invention, the real-time video stream Input Vision Encoder , through the projection layer to get the visual input aligned with the large model , together with the text input obtained previously Enter the Big Language Model.

[0053] In an embodiment of the present invention, the large language model itself is pre-trained with LongLoRA and has long window support capability, and can accommodate longer API guides and their descriptions. The large language model ultimately outputs text answers through a chain of thoughts, and can output an API to control the machine for navigation.

[0054] refer to Figure 2 , Figure 2 It is a structural schematic diagram of a surgical navigation algorithm based on a multimodal large language model provided by the present invention, wherein the navigation system includes: dynamic API human-computer interaction, long window input, and voice activation operation; Here, the user controls the Large Vision Language Model (LVLM) through voice and obtains multimodal feedback from the LVLM; the surgical equipment sends multimodal input to the LVLM and obtains the navigation and operation output by the LVLM for controlling the machine.

[0055] Through the above steps of the embodiment of the present invention, the user voice input is obtained; the user voice input is resampled to obtain the resampled audio; the resampled audio is converted to Mel spectrum and normalized to obtain the voice features; based on the voice features, the voice intent corresponding to the user voice input is determined; the target application interface text is obtained by performing similarity retrieval in the preset data document based on the voice intent through the retriever, wherein the data document includes the application interface document and the application interface manual; the target application interface text is spliced ​​with the voice intent to obtain the text input; the text input and the visual input are input into the pre-trained large language model together to obtain the text answer output by the pre-trained large language model, wherein the text answer is used to control the preset machine for surgical navigation. It can effectively improve the navigation accuracy and response speed in surgery and reduce the application difficulty, thereby solving the defect of high operation difficulty in the surgical navigation method in the prior art.

[0056] According to a surgical navigation method based on a multimodal large language model provided by the present invention, the audio is resampled for Mel spectrum conversion and normalization to obtain speech features, including: Perform Mel spectrum conversion based on user voice input and resampled audio to obtain Mel spectrum; Determine the average value of the Mel-spectrogram over the training dataset; Normalization is performed based on the Mel spectrum and the average value to obtain the normalized speech features.

[0057] Resample the audio ( ) is converted into a Mel spectrum and normalized and input into the speech encoder for command alignment. The specific operations are as follows: First, perform the Mel spectrum conversion operation. For details, please refer to the following formula (3): (3)

[0058] in, represents the Mel spectrum, represents the Mel spectrum conversion operation, Indicates user voice input. Represents resampled audio, Indicates the number of channels, Indicates the window size, Indicates cadence.

[0059] Secondly, the obtained Mel spectrum is normalized. For details, please refer to the following formula (4): (4)

[0060] in, represents the normalized speech features, represents the Mel spectrum, Represents the average value of the Mel-spectrogram on the training dataset.

[0061] Through the embodiments of the present invention, the Mel spectrum conversion converts the spectrum representation of the audio signal to the Mel frequency scale, which is closer to the auditory perception characteristics of the human ear; normalization can eliminate the impact caused by the dimensional difference between different audio signals, making subsequent processing more fair and accurate.

[0062] According to a surgical navigation method based on a multimodal large language model provided by the present invention, based on speech features, a speech intention corresponding to a user's speech input is determined, comprising: Inputting the speech feature into a speech encoder to obtain speech coding features output by the speech encoder; The speech coding features and the preset instruction set are input into the alignment network to obtain the speech intention output by the alignment network, where the speech intention corresponds to the user speech input.

[0063] In the embodiment of the present invention, the obtained normalized speech features are input into the speech encoder, and the extracted speech coding features are aligned with the pre-stored instruction set. For details, refer to the following formula (5): (5)

[0064] in, Indicates the voice intention. represents the aligned network, represents the speech encoder, Represents the normalized speech features.

[0065] Here, the alignment network first extracts meaningful features from the speech signal, such as Mel-frequency cepstral coefficients (MFCC), which can reflect the spectral characteristics and temporal dynamic characteristics of speech. Then, the alignment network is trained using labeled speech and text data. During the training process, the alignment network learns how to align speech features with their corresponding text content. In the inference phase, the alignment network accepts new speech input and outputs results aligned with the text content. This alignment result is usually expressed as the temporal correspondence between text and speech features.

[0066] Through the embodiments of the present invention, in speech recognition, the alignment network can align the obtained speech features according to their intentions, thereby achieving accurate analysis and understanding of the speech signal.

[0067] According to a surgical navigation method based on a multimodal large language model provided by the present invention, before inputting text input and visual input into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, the method further includes: Get live video stream; Input the real-time video stream into the visual encoder to obtain the video features output by the visual encoder; The video features are input into the projection layer to obtain the visual input output by the projection layer.

[0068] In an embodiment of the present invention, a real-time video stream is input into a visual encoder, and a visual input aligned with a large model is obtained through a projection layer, and the visual input is input into a large language model together with the previously obtained text input.

[0069] In an embodiment of the present invention, the visual encoder is typically a deep learning model, such as a convolutional neural network (CNN), which can extract target features from input video frames, where the target features include edges, textures, colors, shapes, etc., or higher-level semantic information (such as object categories, actions, etc.).

[0070] Each frame of the real-time video stream is input into the visual encoder, which outputs the corresponding (video) feature vector or feature map. Video features represent an abstract representation of the video content for subsequent processing or analysis.

[0071] In the embodiment of the present invention, the projection layer can be a transformation layer, such as a fully connected layer, a convolutional layer or a pooling layer, or a more complex network structure, such as a recurrent neural network (RNN) or a self-attention mechanism. The projection layer is used to convert the visual features extracted from the visual encoder into the required visual input format (i.e., video input).

[0072] After the video features are input into the projection layer, they go through a series of transformations to obtain the final visual input, which can be data for further analysis (such as classification, detection, tracking, etc.) or data for generating visual effects (such as augmented reality, virtual reality, etc.).

[0073] According to a surgical navigation method based on a multimodal large language model provided by the present invention, before the text input and the visual input are inputted into the pre-trained large language model together to obtain the text answer outputted by the pre-trained large language model, the method further comprises: The preset large language model is trained based on LongLoRA to obtain a pre-trained large language model.

[0074] In an embodiment of the present invention, the preset large language model itself is pre-trained through LongLoRA, has long window support capability, and can accommodate longer API guides and their descriptions. The large language model ultimately outputs text answers through a chain of thoughts, and can output an API to control the machine for navigation.

[0075] Here, LongLoRA is a pre-training technology optimized for long text or long sequence data. Traditional language models may encounter problems such as loss of contextual information or limited computing resources when processing long text. LongLoRA enables the model to better capture information in long text by improving the model structure or training strategy; having long window support means that the model can handle long text inputs containing a lot of information, such as the complete API guide and its detailed description.

[0076] In the embodiment of the present invention, combined with the above embodiment, the entire multimodal large language model is optimized. Specifically, the training is divided into two stages. The first stage is visual pre-training, aligning the visual and text ends. The second stage is unified instruction fine-tuning training, which uses real data sets to train so that the model can call APIs to control the machine for navigation.

[0077] According to a surgical navigation method based on a multimodal large language model provided by the present invention, text input and visual input are input into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, including: Generate thought chains based on text input and visual input through a pre-trained large language model; Output the target operation application interface and target operation parameters based on the thought chain through the pre-trained large language model; Generate text answers based on the target operation application interface and target operation parameters through a pre-trained large language model.

[0078] In the embodiment of the present invention, the large language model receives inputs from different modalities, including textual modalities (including target application interface text and voice intent) and visual modalities (including video data obtained from real-time video streams). These inputs together constitute the context that the large language model needs to understand and process.

[0079] Here, the large language model uses the knowledge and reasoning ability acquired in pre-training to integrate the input textual information and visual information to form a series of internal reasoning steps or thinking chains, where the thinking chains are used to parse, understand, associate and reason about the input information to build a comprehensive understanding of the input content.

[0080] After the large language model understands the input content, it determines the target operation application program interface (API) to be called, and generates target operation parameters based on the input information and internal reasoning process (thinking chain).

[0081] For example, based on the understanding of API functions and matching the input information with the API functions, the most matched API and target operation parameters are obtained to perform the required tasks. After the target operation API and target operation parameters are generated, the large language model can call the API through internal mechanisms or external interfaces. After the API call is completed, the large language model needs to parse the returned results. Finally, the large language model will generate an easy-to-understand text answer based on the parsed results and input information. The text answer includes the results of the API call, an explanation of the results, and any relevant suggestions or instructions.

[0082] refer to Figure 3 , Figure 3 It is a model schematic diagram of the surgical navigation algorithm based on the multimodal large language model provided by the present invention.

[0083] Here, the doctor inputs the voice command, the voice activation operation module receives the voice command and converts it into a voice mel-spectrogram, which is then input into the voice encoder, and the obtained voice features are aligned with the preset instruction set to obtain the voice intent; the voice intent is input into the dynamic API information retrieval enhancement module, and the preset API document / guide is searched based on the voice intent to obtain the target application program interface document (Top-K API) closest to the target number, and the command input is fused with the target application program interface document and the target application program interface text (API-Former) to obtain the text token (text input); the real-time video stream in the multimodal input is input into the visual encoder, and after the projection layer, the visual token (visual input) is obtained; the text token and the visual token are input into the pre-trained large language model in the long window input module to obtain the API combination output of the large language model, and the API combination output is used to control the navigation and operation of the preset machine.

[0084] Among them, the long window input module includes the large visual language model (LVLM) and dynamic execution. The large visual language model includes the large language model (LLM). The large language model includes the encoding process (Embedding), normalization, multi-head self-attention mechanism (Multi-Head Self-Attention), normalization and feedforward. Here, the input of the multi-head self-attention mechanism includes navigation instructions (auxiliary description, extraction API and extraction guide).

[0085] The following describes an example of a surgical navigation method based on a multimodal large language model provided by the present invention in a practical application scenario.

[0086] The core of the embodiment of the present invention is to solve the problem of accurate navigation of deep brain structures during surgery in the field of neurosurgery. In view of the limitations of existing navigation technology, including over-reliance on experienced neurosurgeons and lack of accuracy and operational consistency, the present invention proposes an intelligent navigation system based on a multimodal large language model (LVLM). The system integrates preoperative MRI scans, real-time sensor feedback, text instructions from the surgical team, and intraoperative visual data streams. Through efficient visual encoder and text decoder technology, it optimizes the representation of visual and text data from the bottom layer to achieve in-depth integration of cross-modal knowledge.

[0087] Specifically, by combining the mask autoencoder task of the visual modality with the next token prediction task of the text modality, the system can not only fully understand and integrate medical images and direct instructions, but also effectively improve the navigation accuracy and response speed during surgery through a joint generative pre-training strategy. Overall, the intelligent navigation system of the present invention significantly improves the accuracy and application efficiency of surgery, has a wide range of application potential, and provides a new technical solution for future neurosurgery.

[0088] The present invention comprises the following steps: Step S0, receiving the doctor's voice As voice input; Step S1, resample the audio to obtain resampled audio ; Step S2, resample the audio Convert to Mel spectrum, normalize, and input into speech encoder for command alignment. The specific operations are as follows: First, perform the Mel spectrum conversion operation. For details, please refer to the following formula (6): (6)

[0089] in, represents the Mel spectrum, represents the Mel spectrum conversion operation, Indicates user voice input. Represents resampled audio, Indicates the number of channels, Indicates the window size, Indicates cadence.

[0090] Secondly, the obtained Mel spectrum is normalized. For details, please refer to the following formula (7): (7)

[0091] in, represents the normalized speech features, represents the Mel spectrum, Represents the average value of the Mel-spectrogram on the training dataset.

[0092] Finally, the normalized speech features are input into the speech encoder, and the extracted speech coding features are aligned with the pre-stored instruction set. For details, please refer to the following formula (8): (8)

[0093] in, Indicates the voice intention. represents the aligned network, represents the speech encoder, Represents the normalized speech features.

[0094] Step S3: perform similarity search in the pre-stored API documents and manuals through the search engine R according to the obtained intent C, and obtain the closest API and its description T. API , please refer to the following formula (9): (9)

[0095] in, Represents the target application interface text, The function is used to sequentially select the target number of elements (with the greatest similarity) from a set or array. Represents a retriever, Indicates application program interface document and application program interface manual.

[0096] Step S4, the obtained target application program interface text ( ) and voice intent ( ) collection to obtain the text input of the large language model ( ), please refer to the following formula (10): (10)

[0097] in, represents the text input of the large language model, Represents a splicing operation, Represents the target application interface text, Indicates voice intent.

[0098] Step S5: input the real-time video stream V into the visual encoder V E , through the projection layer to obtain the visual input V aligned with the large model input , together with the previously obtained text input T input Enter the Big Language Model.

[0099] Step S6, the large language model itself is pre-trained through LongLoRA and has long window support capabilities, which can accommodate longer API guides and their descriptions. The large language model finally outputs text answers through the thought chain and can output API to control the machine for navigation.

[0100] The surgical navigation device based on a multimodal large language model provided by the present invention is described below. The surgical navigation device based on a multimodal large language model described below and the surgical navigation method based on a multimodal large language model described above can be referenced to each other.

[0101] refer to Figure 4 , Figure 4 It is a structural diagram of a surgical navigation device based on a multimodal large language model provided by the present invention, which includes an acquisition module 401, a resampling module 402, a conversion module 403, a determination module 404, a retrieval module 405, a splicing module 406, and an output module 407.

[0102] The acquisition module 401 is used to acquire the user's voice input; The resampling module 402 is used to resample the user's voice input to obtain a resampled audio; The conversion module 403 is used to perform Mel spectrum conversion and normalization on the resampled audio to obtain speech features; A determination module 404, configured to determine a speech intention corresponding to a user speech input based on speech features; A retrieval module 405 is used to perform similarity retrieval in a preset data document based on the speech intent through a retriever to obtain a target application interface text, wherein the data document includes an application interface document and an application interface manual; A splicing module 406, configured to splice the target application program interface text with the voice intent to obtain a text input; The output module 407 is used to input the text input and the visual input into the pre-trained large language model to obtain the text answer output by the pre-trained large language model, wherein the text answer is used to control the preset machine for surgical navigation.

[0103] Specifically, the surgical navigation device based on a multimodal large language model provided by the present invention can implement all the method steps implemented in the above-mentioned surgical navigation method embodiment based on a multimodal large language model, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0104] Figure 5 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a surgical navigation method based on a multimodal large language model, the method comprising: obtaining a user voice input; resampling the user voice input to obtain a resampled audio; performing Mel spectrum conversion and normalization on the resampled audio to obtain a voice feature; determining a voice intent corresponding to the user voice input based on the voice feature; performing a similarity search in a preset data document based on the voice intent through a retriever to obtain a target application interface text, wherein the data document includes an application interface document and an application interface manual; splicing the target application interface text with the voice intent to obtain a text input; inputting the text input and the visual input into a pre-trained large language model together to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine for surgical navigation.

[0105] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the surgical navigation method based on a multimodal large language model provided by the above methods, the method including: obtaining user voice input; resampling the user voice input to obtain resampled audio; performing Mel spectrum conversion and normalization on the resampled audio to obtain voice features; based on the voice features, determining the voice intent corresponding to the user voice input; performing similarity search in a preset data document based on the voice intent through a retriever to obtain a target application interface text, wherein the data document includes an application interface document and an application interface manual; splicing the target application interface text with the voice intent to obtain a text input; inputting the text input and the visual input together into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine for surgical navigation.

[0107] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to execute the surgical navigation method based on the multimodal large language model provided by the above methods, the method comprising: obtaining user voice input; resampling the user voice input to obtain resampled audio; performing Mel spectrum conversion and normalization on the resampled audio to obtain voice features; based on the voice features, determining the voice intent corresponding to the user voice input; performing similarity retrieval in a preset data document based on the voice intent through a retriever to obtain a target application interface text, wherein the data document includes an application interface document and an application interface manual; splicing the target application interface text with the voice intent to obtain a text input; inputting the text input and the visual input together into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control the preset machine for surgical navigation. The above-described device embodiments are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those skilled in the art may understand and implement the solution without creative work.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surgical navigation method based on a multimodal large language model, characterized in that: include: Get user voice input; Resampling the user voice input to obtain resampled audio; Performing Mel spectrum conversion and normalization on the resampled audio to obtain speech features; Based on the voice feature, determining a voice intention corresponding to the user voice input; A searcher is used to perform similarity search in a preset data document based on the speech intent to obtain a target application program interface text, wherein the data document includes an application program interface document and an application program interface manual; Based on the target application program interface text and the voice intention, a text input is obtained by concatenating the text with the voice intention; The text input and the visual input are input together into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine for surgical navigation.

2. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that: The step of performing Mel spectrum conversion and normalization on the resampled audio to obtain speech features includes: Performing Mel spectrum conversion based on the user voice input and the resampled audio to obtain a Mel spectrum; Determine an average value of the Mel spectrum on the training data set; Normalization is performed based on the Mel spectrum and the average value to obtain normalized speech features.

3. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that: The determining, based on the voice feature, a voice intention corresponding to the user voice input includes: Inputting the speech feature into a speech encoder to obtain speech coding features output by the speech encoder; The speech coding features and a preset instruction set are input into an alignment network to obtain a speech intention output by the alignment network, wherein the speech intention corresponds to the user speech input.

4. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that: Before inputting the text input and the visual input together into the pre-trained large language model to obtain a text answer output by the pre-trained large language model, the method further includes: Get live video stream; Inputting the real-time video stream into a visual encoder to obtain video features output by the visual encoder; The video features are input into the projection layer to obtain the visual input output by the projection layer.

5. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that: Before inputting the text input and the visual input together into the pre-trained large language model to obtain a text answer output by the pre-trained large language model, the method further includes: The preset large language model is trained based on LongLoRA to obtain a pre-trained large language model.

6. The surgical navigation method based on a multimodal large language model according to claim 1, characterized in that: The step of inputting the text input and the visual input into a pre-trained large language model to obtain a text answer output by the pre-trained large language model includes: Generate a thought chain based on the text input and the visual input through a pre-trained large language model; Outputting a target operation application program interface and target operation parameters based on the thought chain through the pre-trained large language model; Generate a text answer based on the target operation application program interface and the target operation parameters through the pre-trained large language model.

7. A surgical navigation device based on a multimodal large language model, characterized in that: include: An acquisition module is used to acquire user voice input; A resampling module, used to resample the user voice input to obtain resampled audio; A conversion module, used for performing Mel spectrum conversion and normalization on the resampled audio to obtain speech features; A determination module, configured to determine a voice intention corresponding to the user voice input based on the voice feature; A retrieval module, used to perform similarity retrieval in a preset data document based on the speech intent through a retriever to obtain a target application interface text, wherein the data document includes an application interface document and an application interface manual; A splicing module, used for splicing the target application program interface text with the voice intention to obtain a text input; An output module is used to input the text input and the visual input into a pre-trained large language model to obtain a text answer output by the pre-trained large language model, wherein the text answer is used to control a preset machine for surgical navigation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the surgical navigation method based on the multimodal large language model is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the surgical navigation method based on a multimodal large language model is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the surgical navigation method based on a multimodal large language model is implemented as described in any one of claims 1 to 6.

Images