A visual navigation method, device and medium for a medical embodied robot

Abstract

The present invention relates to the technical field of embodied medical robots, and specifically to a visual navigation method, device, and medium for a medical embodied robot, the method comprising the following steps: S1, obtaining a unified feature vector; S2, constructing an initial model; S3, optimizing the initial model; and S4, completing visual navigation. The main technical problem addressed by the present invention is the real-time visual navigation and multi-task adaptability of medical embodied intelligent robots in complex medical scenarios, particularly the fusion and processing of multimodal data in resource-constrained environments. By optimizing the neural architecture and adopting delay control and multimodal data alignment technology, the present invention improves the real-time performance and accuracy of the robot in a surgical environment, ensuring that it can handle different tasks and adapt to environmental changes in dynamic scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of embodied medical robots, and in particular to a visual navigation method, device and medium for an embodied medical robot. Background Art

[0002] The field of embodied intelligence is developing rapidly, and the application of embodied intelligent robots in medicine is also gradually increasing. These robots are particularly important in scenarios such as surgical assistance, precision diagnosis, rehabilitation care, and telemedicine. Medical embodied robots require high-precision visual navigation capabilities to achieve autonomous movement, environmental perception, and task execution. For example, in the operating room, robots must not only accurately identify the position of surgical tools and organs, but also adjust their operating paths in real time to avoid interference with doctors and other equipment, ensuring a safe and efficient surgical process. Therefore, the medical environment places higher demands on the robot's visual navigation in terms of accuracy, real-time performance, and safety.

[0003] Traditional visual navigation technology faces the following major challenges in medical scenarios:

[0004] Multimodal Data Processing: The medical environment involves diverse data types (e.g., real-time images, CT / MRI scans, and patient physiological data). The effective fusion and real-time processing of this data are crucial for robotic navigation and diagnostic capabilities. In medical imaging, particularly CT and MRI scan processing, multimodal data fusion is required to improve diagnostic accuracy.

[0005] Adaptability to complex environments: There are many interference factors in medical scenarios, such as light changes, obstructions, and dynamic objects (such as the movement of medical staff), which can affect the robot's visual navigation effect.

[0006] Real-time and resource constraints: Medical embodied robots require fast responses, and visual navigation tasks place high demands on computing resources. Therefore, an efficient and lightweight neural architecture is urgently needed.

[0007] High precision and robustness requirements: The error tolerance in medical application scenarios is extremely low, and robots need to have high-precision and robust visual navigation capabilities to ensure safety and effectiveness.

[0008] In summary, there is an urgent need for a visual navigation method, device and medium for medical embodied robots to solve the problems in the existing technology. Summary of the Invention

[0009] The present invention aims to provide a visual navigation method, device, and medium for a medical embodied robot. The specific technical solutions are as follows:

[0010] A visual navigation method for a medical embodied robot comprises the following steps:

[0011] S1. Obtaining a unified feature vector: Obtain multimodal data, preprocess, align, and weightedly fuse the multimodal data to obtain a unified feature vector;

[0012] S2. Build an initial model: Use neural architecture search to generate a neural network architecture for visual navigation. Based on the neural network architecture, add lightweight convolution and attention mechanisms to obtain the initial model.

[0013] S3. Optimize the initial model: Construct an optimization objective function based on delay regularization, optimize the initial model based on the optimization objective function, and obtain a visual navigation model;

[0014] S4. Complete visual navigation: Input the unified feature vector into the visual navigation model to realize visual navigation of the medical embodied robot.

[0015] Optionally, in S1, the multimodal data includes real-time camera images, medical images and patient physiological data.

[0016] Optionally, in S1, the multimodal data is preprocessed, and the preprocessing includes adaptive histogram equalization and median filtering, the adaptive histogram equalization is used to enhance contrast, and the median filtering is used to remove noise.

[0017] Optionally, in S1, the multimodal data is aligned, including spatial alignment and temporal alignment;

[0018] The steps for spatial alignment are as follows:

[0019] Feature point extraction: Extract key feature points from real-time camera images and medical images. For real-time camera images, edge detection algorithms are used to find identifiable boundaries or reference points in the environment as key feature points. For medical images, edge detection algorithms or feature extraction algorithms are used to extract anatomical landmarks as key feature points.

[0020] Feature point matching: Use matching algorithms to match key feature points in real-time camera images and medical images to obtain matched key feature point pairs;

[0021] Estimate the spatial transformation matrix: Use the matched key feature point pairs to calculate the spatial transformation matrix applicable to the two modes, and estimate the spatial transformation matrix parameters using the least squares method;

[0022] Image resampling and registration: Based on the calculated spatial transformation matrix, the medical image is resampled so that the medical image is presented in the same coordinate system as the real-time camera image, completing spatial alignment;

[0023] The steps for time alignment are as follows:

[0024] Determine the sampling frequency: obtain the sampling frequency of each modal data;

[0025] Timestamp alignment: annotate each modality data with an accurate timestamp;

[0026] Interpolation processing: Apply interpolation algorithms to low-frequency data to make the time steps of low-frequency data the same as those of high-frequency data;

[0027] Time synchronization processing: For high-frequency data, downsampling is performed according to the sampling period of low-frequency data to ensure that the time points of each modal data correspond one-to-one and complete time alignment.

[0028] Optionally, in S1, weighted fusion is performed on the multimodal data, and the expression is as follows:

[0029] ;

[0030] in, represents the unified eigenvector, represents the aligned real-time camera image, represents the weight coefficient of the real-time camera image, represents the aligned medical image, represents the weight coefficient of medical imaging, represents the aligned patient physiological data, Represents the weight coefficient of the patient's physiological data; 、 、 Adaptive adjustment through model training.

[0031] Optionally, in S2, the initial model includes a feature extraction layer, an attention mechanism, a multi-task adaptation layer, and a path planning module, which are arranged in sequence;

[0032] The feature extraction layer is a depth-separable convolutional layer for feature extraction;

[0033] The attention mechanism is a channel-level attention mechanism, which is used to process the extracted features;

[0034] The multi-task adaptation layer includes multiple branch networks, each branch network corresponding to a different type of task;

[0035] The path planning module outputs path planning based on the unified feature vector to complete visual navigation.

[0036] Optionally, in S3, the expression of the optimization objective function is as follows:

[0037] ;

[0038] in, Represents the model prediction value and the true label value The error represents the loss function; Indicates that the model has parameters and neural network architecture Lower pair input The predicted output of represents the input data of the sample, represents the total latency of the neural network architecture, represents the neural network architecture, represents the total number of samples, represents the regularization coefficient.

[0039] Optionally, the total latency of the neural network architecture in S3 The expression is as follows:

[0040] ;

[0041] in, Indicates candidate operations The average execution time of Indicates candidate operations In neural network architecture The probability of selection under It is calculated by the softmax function, and the calculation expression is as follows:

[0042] ;

[0043] in, Indicates the candidate operation in the architecture parameter The relevant weights, Represents the normalization factor, which is used to normalize the probability and the sum of candidate operations.

[0044] Additionally, a computer device includes a memory and a processor;

[0045] The memory is used to store a computer program that can be executed on the processor;

[0046] The processor is configured to implement the steps of the above-mentioned visual navigation method when executing the computer program.

[0047] In addition, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the visual navigation method described above are implemented.

[0048] The application of the technical solution of the present invention has the following beneficial effects:

[0049] (1) This invention discloses a visual navigation method for an embodied medical robot. Unlike convolutional neural networks (CNNs), which have a fixed structure and are difficult to handle in dynamic and multimodal scenarios, this method ensures real-time geometric and temporal consistency by spatially and temporally aligning multimodal data. Furthermore, this method fuses multimodal data through an adaptive weighting mechanism, further enhancing its adaptability to dynamic changes. This enables the embodied medical robot to effectively extract key features from multi-scale and multimodal data, more accurately adapting to the real-time needs of complex medical scenarios.

[0050] (2) Traditional multimodal data fusion methods have high computational complexity and are prone to misjudgment in real-time applications. The method of the present invention discloses spatial alignment and temporal alignment strategies to accurately synchronize the data of each modality, reduce the computational burden, and shorten data processing time. The method of the present invention can effectively avoid feature conflicts and ensure feature fusion and rapid response in complex dynamic environments.

[0051] (3) The method of the present invention discloses an optimization objective function based on delay regularization. Through delay optimization control, a low-latency and efficient neural network architecture is selected, reducing the computational overhead of the model. Compared with traditional neural architecture search (NAS), the method of the present invention reduces latency without compromising performance, ensuring that the medical embodied robot can achieve stable path navigation and rapid response in multi-tasking real-time medical scenarios.

[0052] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 It is a flowchart of the steps of the visual navigation method in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] like Figure 1 As shown, this embodiment provides a visual navigation method for a medical embodied robot, comprising the following steps:

[0057] S1. Obtaining a unified feature vector: Obtain multimodal data, preprocess, align, and weightedly fuse the multimodal data to obtain a unified feature vector;

[0058] S2. Build an initial model: Use neural architecture search to generate a neural network architecture for visual navigation. Based on the neural network architecture, add lightweight convolution and attention mechanisms to obtain the initial model.

[0059] S3. Optimize the initial model: Construct an optimization objective function based on delay regularization, optimize the initial model based on the optimization objective function, and obtain a visual navigation model;

[0060] S4. Complete visual navigation: Input the unified feature vector into the visual navigation model to realize visual navigation of the medical embodied robot.

[0061] Specifically, in S1, the multimodal data includes real-time camera images, medical images, and patient physiological data. In this embodiment, the real-time camera images are used for obstacle detection and path navigation, helping the robot plan a safe path in a dynamic environment; the medical images include CT images or MRI images, which are used to identify specific parts such as target organs, lesions, and surgical instruments to improve the accuracy of diagnosis and navigation; the patient's physiological data, including heart rate, blood pressure, and respiratory rate, serve as auxiliary judgments to help the robot identify the patient's current condition and make adaptive adjustments.

[0062] Specifically, in S1, the multimodal data is preprocessed, and the preprocessing includes adaptive histogram equalization and median filtering. The adaptive histogram equalization is used to enhance contrast, and the median filtering is used to remove noise.

[0063] Specifically, the adaptive histogram equalization (CLAHE) method is as follows:

[0064] Since medical images usually have low contrast, CLAHE is used to enhance the contrast of the image to make the details clearer. CLAHE limits the contrast improvement when processing each local area to prevent global over-enhancement and image noise. The CLAHE enhancement formula is as follows:

[0065] ;

[0066] in, and Represent the minimum grayscale value and maximum grayscale value of the original image respectively. represents the original pixel value, represents the pixel value after image enhancement, Indicates the minimum pixel value.

[0067] In this embodiment, CLAHE can enhance the details of low-contrast areas while preserving the overall features of the image, which helps improve the accuracy in subsequent recognition tasks.

[0068] Optionally, in S1, the multimodal data is aligned, including spatial alignment and temporal alignment;

[0069] The steps for spatial alignment are as follows:

[0070] Feature point extraction: Extract key feature points from real-time camera images and medical images. For real-time camera images, edge detection algorithms are used to find identifiable boundaries or reference points in the environment as key feature points. For medical images, edge detection algorithms or feature extraction algorithms are used to extract anatomical landmarks as key feature points.

[0071] Feature point matching: Use matching algorithms to match key feature points in real-time camera images and medical images to obtain matched key feature point pairs;

[0072] Estimate the spatial transformation matrix: Use the matched key feature point pairs to calculate the spatial transformation matrix applicable to the two modes, and estimate the spatial transformation matrix parameters using the least squares method;

[0073] Image resampling and registration: Based on the calculated spatial transformation matrix, the medical image is resampled so that the medical image is presented in the same coordinate system as the real-time camera image, completing spatial alignment;

[0074] The steps for time alignment are as follows:

[0075] Determine the sampling frequency: obtain the sampling frequency of each modal data;

[0076] Timestamp alignment: annotate each modality data with an accurate timestamp;

[0077] Interpolation: Applying an interpolation algorithm to low-frequency data (such as patient physiological data) to make the time step of low-frequency data the same as that of high-frequency data (such as real-time camera images);

[0078] The interpolation method in this embodiment includes at least one of the following:

[0079] Linear interpolation: suitable for small sampling frequency differences;

[0080] Spline Interpolation: When data sampling is sparse and high-precision alignment is required, cubic spline interpolation can be used to smooth the data;

[0081] The interpolation formula for physiological data is:

[0082] ;

[0083] in, and For two adjacent time points, is the interpolated data value.

[0084] Time synchronization processing: For high-frequency data, downsampling is performed according to the sampling period of low-frequency data to ensure that the time points of each modal data correspond one-to-one and complete time alignment.

[0085] It should be noted that this embodiment also includes verifying and optimizing the alignment results after spatial alignment or temporal alignment. The verification of spatial alignment is performed by superimposing two images, and the verification of temporal alignment is performed by checking the temporal alignment effect of the interpolated data.

[0086] Optionally, in S1, weighted fusion is performed on the multimodal data, and the expression is as follows:

[0087] ;

[0088] in, represents the unified eigenvector, represents the aligned real-time camera image, represents the weight coefficient of the real-time camera image, represents the aligned medical image, represents the weight coefficient of medical imaging, represents the aligned patient physiological data, Represents the weight coefficient of the patient's physiological data; 、 、 Adaptive adjustment through model training.

[0089] Optionally, in S2, the initial model includes a feature extraction layer, an attention mechanism, a multi-task adaptation layer, and a path planning module, which are arranged in sequence;

[0090] The feature extraction layer is a depth-separable convolutional layer for feature extraction;

[0091] It should be noted that for a depth-wise separable convolution operation, its calculation can be divided into two steps: depth-wise convolution and point-wise convolution:

[0092] Depth convolution: Each channel is convolved independently to extract spatial features without crossing channels. The depth convolution calculation formula is:

[0093] ;

[0094] in: represents the output feature map after depth convolution, Represents a depth convolution kernel that operates independently on each channel; Represents the position in the input feature map Pixel value of

[0095] Pointwise Convolution:

[0096] Use 1×1 convolution operation to perform convolution across channels to fuse information from different channels. The point-by-point convolution calculation formula is:

[0097] ;

[0098] in, Represents the output feature map after point-by-point convolution, It is a point-by-point convolution kernel used to fuse the features of each channel of the depth convolution. Compared with traditional convolution, the computational complexity of depth-wise separable convolution is significantly reduced.

[0099] The attention mechanism is a channel-level attention mechanism (SE module) used to process the extracted features;

[0100] The multi-task adaptation layer includes multiple branch networks, each branch network corresponds to a different type of task. The branch networks in this embodiment include instrument identification branches, lesion localization branches, and path planning branches, etc. This embodiment can further divide the branch networks according to the usage scenarios. This embodiment realizes differentiated needs in medical scenarios by assigning different tasks to different branch networks.

[0101] It should be noted that the path planning branch in this embodiment uses deep reinforcement learning (DRL) to continuously interact with the environment and conduct feedback learning, so that the robot gradually masters the optimal path planning strategy. In DRL, the robot's path planning can be expressed as a Markov decision, and the state, action, and reward settings are as follows:

[0102] State s: represents the current state of the robot in the surgical environment, including information such as position, direction, obstacle positions in the environment, and target position.

[0103] Action a: represents the movement direction or behavior that the robot can choose in state s, such as forward, left, right, etc.

[0104] Reward r: Each time the robot completes an action, it receives a reward. The size of the reward depends on whether the action helps it approach the goal. For example, if the robot successfully avoids an obstacle and moves toward the goal, the reward is positive; if it hits an obstacle, the reward is negative.

[0105] The path planning module outputs path planning based on the unified feature vector to complete visual navigation.

[0106] Optionally, in S3, the expression of the optimization objective function is as follows:

[0107] ;

[0108] in, Represents the model prediction value and the true label value The error represents the loss function; Indicates that the model has parameters and neural network architecture Lower pair input The predicted output of represents the input data of the sample, represents the total latency of the neural network architecture, represents the neural network architecture, represents the total number of samples, represents the regularization coefficient.

[0109] Optionally, the total latency of the neural network architecture in S3 The expression is as follows:

[0110] ;

[0111] in, Indicates candidate operations The average execution time of Indicates candidate operations In neural network architecture The probability of selection under It is calculated by the softmax function, and the calculation expression is as follows:

[0112] ;

[0113] in, Indicates the candidate operation in the architecture parameter The relevant weights, Represents the normalization factor, which is used to normalize the probability and the sum of candidate operations.

[0114] Furthermore, in this embodiment, the DARTS (differentiable search) search space is preferably used, and its search space setting and architecture parameter optimization are as follows:

[0115] DARTS's search space contains multiple candidate operations (such as convolution, pooling, and skip connections). Different combinations of these operations affect the accuracy and latency of the model. The architecture parameters are optimized by continuously refining the search space. This process is divided into multiple steps to ensure that the resulting architecture meets the adaptability requirements of multi-task environments.

[0116] Search space design:

[0117] Convolution operation: including different convolution kernel sizes and strides.

[0118] 3x3 convolution: Suitable for fine feature extraction, helpful for tasks such as lesion detection.

[0119] 5x5 convolution: used for feature extraction with a larger receptive field, suitable for global feature recognition.

[0120] Pooling operations: including maximum pooling and average pooling.

[0121] Max pooling: Strengthens the highlights in the feature map, which is suitable for highlighting important features in lesion detection tasks.

[0122] Average pooling: smooths feature maps, reduces noise, and helps extract background information.

[0123] Stride selection: Stride 1 maintains the original resolution, and stride 2 is used for downsampling to reduce the amount of computation.

[0124] Neural network architecture parameter optimization:

[0125] DARTS controls the selection of operations through architectural parameters. The architectural parameters of each layer are initialized to a uniform distribution and gradually optimized during the search process. The specific steps are as follows:

[0126] 1) Initialize architecture parameters: At the beginning of the search, randomly initialize the architecture parameters α of each candidate operation.

[0127] 2) Calculation of operation selection probability: For each node, DARTS calculates the selection probability of the operation through the softmax function:

[0128] ;

[0129] 3) Continuation of hybrid operation: DARTS makes the discrete search space continuous through hybrid operation. The hybrid operation formula is as follows:

[0130] ;

[0131] in, Indicates an operation Input Output; Indicates in and The mixed operation output between nodes contains the weighted sum of all candidate operations, Indicates selection operation probability.

[0132] Gradient Optimization: DARTS uses a dual optimization process to optimize weight parameters and architecture parameters separately.

[0133] Model weight optimization: Use the standard gradient descent algorithm to update the model weights to reduce the loss function.

[0134] Architecture parameter optimization: The gradient of the architecture parameters is calculated through chain derivation, and the architecture parameters are updated to select a better operation combination.

[0135] In addition, this embodiment also provides a computer device, including a memory and a processor;

[0136] The memory is used to store a computer program that can be executed on the processor;

[0137] The processor is used to implement the steps of the above-mentioned visual navigation method when executing the computer program.

[0138] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0139] The computer device may be a mobile phone, desktop computer, laptop, PDA, cloud server, or other computing device. The computer device may include, but is not limited to, a processor and memory. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0140] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor is the control center of the computer device and connects the various parts of the entire computer device using various interfaces and lines.

[0141] The memory can be used to store the computer programs and / or modules. The processor implements the computer programs by running or executing the computer programs and / or modules stored in the memory and accessing data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of the mobile phone (such as audio data, a phone book, etc.). Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0142] If the module / unit integrated into the computer device is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0143] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned visual navigation method are implemented.

[0144] This embodiment method discloses a visual navigation method for an embodied medical robot. Unlike convolutional neural networks (CNNs), which have a fixed structure and are difficult to handle in dynamic and multimodal scenarios, this embodiment method ensures real-time geometric and temporal consistency by spatially and temporally aligning multimodal data. Furthermore, this embodiment method fuses multimodal data through an adaptive weighting mechanism, further enhancing its adaptability to dynamic changes. This enables the embodied medical robot to effectively extract key features from multi-scale and multimodal data, more precisely adapting to the real-time needs of complex medical scenarios.

[0145] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0146] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A computer-readable storage medium, characterized in that The computer readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a visual navigation method for a medical embodied robot; The visual navigation method of the medical embodied robot includes the following steps: S1. Obtaining a unified feature vector: Obtain multimodal data, preprocess, align, and weightedly fuse the multimodal data to obtain a unified feature vector; S2. Build an initial model: Use neural architecture search to generate a neural network architecture for visual navigation. Based on the neural network architecture, add lightweight convolution and attention mechanisms to obtain the initial model. S3. Optimize the initial model: Construct an optimization objective function based on delay regularization, optimize the initial model based on the optimization objective function, and obtain a visual navigation model; S4. Complete visual navigation: Input the unified feature vector into the visual navigation model to achieve visual navigation of the medical embodied robot; In S1, the multimodal data are aligned, including spatial alignment and temporal alignment; The steps for spatial alignment are as follows: Feature point extraction: Extract key feature points from real-time camera images and medical images. For real-time camera images, edge detection algorithms are used to find identifiable boundaries or reference points in the environment as key feature points. For medical images, edge detection algorithms or feature extraction algorithms are used to extract anatomical landmarks as key feature points. Feature point matching: Use matching algorithms to match key feature points in real-time camera images and medical images to obtain matched key feature point pairs; Estimate the spatial transformation matrix: Use the matched key feature point pairs to calculate the spatial transformation matrix applicable to the two modes, and estimate the spatial transformation matrix parameters using the least squares method; Image resampling and registration: Based on the calculated spatial transformation matrix, the medical image is resampled so that the medical image is presented in the same coordinate system as the real-time camera image, completing spatial alignment; The steps for time alignment are as follows: Determine the sampling frequency: obtain the sampling frequency of each modal data; Timestamp alignment: annotate each modality data with an accurate timestamp; Interpolation processing: Apply interpolation algorithms to low-frequency data to make the time steps of low-frequency data the same as those of high-frequency data; Time synchronization processing: For high-frequency data, downsampling is performed according to the sampling period of low-frequency data to ensure that the time points of each modal data correspond one-to-one and complete time alignment.

2. The computer-readable storage medium according to claim 1, wherein In S1, the multimodal data includes real-time camera images, medical images and patient physiological data.

3. The computer-readable storage medium according to claim 2, wherein In S1 , the multimodal data is preprocessed. The preprocessing includes adaptive histogram equalization and median filtering. The adaptive histogram equalization is used to enhance contrast, and the median filtering is used to remove noise.

4. The computer-readable storage medium according to claim 3, wherein: In S1, multimodal data is weightedly fused, and the expression is as follows: F fused =α·F cam +β·F CT / MRI +γ·F bio ; Among them, F fused represents the unified eigenvector, F cam represents the aligned real-time camera image, α represents the weight coefficient of the real-time camera image, F CT / MRI represents the aligned medical image, β represents the weight coefficient of the medical image, F bio represents the aligned patient physiological data, γ represents the weight coefficient of the patient physiological data; α, β, and γ are adaptively adjusted through model training.

5. The computer-readable storage medium according to claim 4, wherein: In S2, the initial model includes the feature extraction layer, attention mechanism, multi-task adaptation layer and path planning module set in sequence; The feature extraction layer is a depth-separable convolutional layer for feature extraction; The attention mechanism is a channel-level attention mechanism, which is used to process the extracted features; The multi-task adaptation layer includes multiple branch networks, each branch network corresponding to a different type of task; The path planning module outputs path planning based on the unified feature vector to complete visual navigation.

6. The computer-readable storage medium according to claim 5, wherein: In S3, the expression of the optimization objective function is as follows: Among them, Loss(y i ,p(x i ;W,Θ)) represents the model prediction value p(x i ) and the true label value y i The error represents the loss function; p(x i ;W,Θ) represents the model's response to input x under parameter W and neural network architecture Θ i The predicted output of x i Represents the input data of the sample, Lat(Θ) represents the total delay of the neural network architecture, Θ represents the neural network architecture, n represents the total number of samples, and λ represents the regularization coefficient.

7. The computer-readable storage medium according to claim 6, wherein: In S3, the total latency Lat(Θ) of a neural network architecture is expressed as follows: Lat(Θ)=∑ o∈O Years o ·Prob(o|Θ); Among them, LaT o represents the average execution time of candidate operation o; Prob(o|Θ) represents the selection probability of candidate operation o under the neural network architecture Θ. Prob(o|Θ) is calculated by the softmax function, and the calculation expression is as follows: Where α0 represents the weight of the architecture parameters associated with the candidate operation o, ∑ o′∈O exp(α o′ ) represents the normalization factor, which is used to normalize the probability and value of the candidate operations.

8. A computer device, characterized in that: including memory and processor; The memory is used to store a computer program that can be executed on the processor; The processor is configured to implement the steps of the visual navigation method according to any one of claims 1 to 7 when executing the computer program.

Images