A high-precision tracking and positioning system for dynamic objects based on AR medical navigation

Abstract

The present invention relates to the field of medical navigation technology, and specifically to a high-precision tracking and positioning system for dynamic objects based on AR medical navigation, including a multi-angle imaging module, a micro-dynamic detection module, a data processing and analysis module, a dynamic calibration and synchronization module, and an enhanced visual mapping module; wherein the detailed visual and depth information in the surgical area is captured; the micro-dynamic detection module: captures and records the tiny dynamic changes of the surgical object caused by physiological or mechanical factors; the data processing and analysis module: extracts the position, motion trajectory and behavior pattern of the object; the dynamic calibration and synchronization module: adjusts the view displayed by the AR device; the enhanced visual mapping module: presents the results on the AR device of the medical staff. The present invention, through real-time and accurate dynamic tracking and positioning, combined with the automatically adjusted enhanced visual mapping technology, significantly improves the accuracy and safety of the surgery, optimizes the visual experience, and improves the efficiency of the surgery.

Description

Technical Field

[0001] The present invention relates to the field of medical navigation technology, and in particular to a high-precision tracking and positioning system for dynamic objects based on AR medical navigation. Background Art

[0002] In modern medical surgery, especially in complex surgical operations such as brain surgery or heart surgery, accurate navigation systems are crucial to improving surgical success rates and patient safety. With the advancement of technology, augmented reality (AR) technology has been gradually introduced into operating rooms to display detailed three-dimensional views of the surgical area and the location of surgical tools in real time. However, traditional AR surgical navigation systems often face problems such as delayed data updates, asynchrony between images and actual views, and unclear display of visual information under different lighting conditions. These problems limit the application effect and popularity of AR technology in surgical navigation.

[0003] Although the current surgical navigation system can provide a certain degree of visual assistance, in actual operations, the subtle dynamic changes of the surgical object are often difficult to capture and accurately display in real time, which affects the accuracy of the surgery. In addition, the existing system has shortcomings in dynamic calibration and visual mapping, such as the inability to automatically adjust the brightness, contrast and color of the mapped content, making it difficult for medical staff to obtain the best visual experience in a changing surgical environment.

[0004] Therefore, developing an AR navigation system that can achieve high-precision real-time tracking and positioning and provide optimized visual display is of great significance for improving surgical safety and efficiency and expanding the scope of application of AR technology in the medical field. Summary of the invention

[0005] Based on the above objectives, the present invention provides a high-precision tracking and positioning system for dynamic objects based on AR medical navigation.

[0006] A high-precision tracking and positioning system for dynamic objects based on AR medical navigation, including a multi-angle imaging module, a micro-dynamic detection module, a data processing and analysis module, a dynamic calibration and synchronization module, and an enhanced visual mapping module; wherein:

[0007] Multi-angle imaging module: equipped with high-resolution cameras and depth sensors at multiple angles to capture detailed visual and depth information within the surgical area to provide a three-dimensional view of the surgical object and its surroundings;

[0008] Micro-dynamic detection module: integrated with high-sensitivity vibration sensor to capture and record small dynamic changes of surgical objects caused by physiological or mechanical factors, so as to enhance the tracking details and accuracy;

[0009] Data processing and analysis module: Based on the data from the multi-angle imaging module and the micro-dynamic detection module, the preset image processing algorithm and machine learning technology are used for analysis to extract the position, motion trajectory and behavior pattern of the object;

[0010] Dynamic calibration and synchronization module: adjusts the view displayed by the AR device according to the information provided by the data processing and analysis module to ensure that the position of the surgical object in the AR view is completely synchronized with the position in the actual surgery;

[0011] Enhance the visual mapping module; present the data results of the data processing and analysis module and the dynamic calibration and synchronization module on the medical staff's AR device in the form of visual overlay. During the process, adjust the brightness, contrast and color of the mapping content through visual optimization technology.

[0012] Furthermore, the multi-angle imaging module includes a configuration unit, a synchronization control unit, a data fusion unit and an optimization processing unit; wherein:

[0013] Configuration unit: equipped with high-resolution cameras at at least three angles and at least two depth sensors, the cameras and sensors are evenly spaced around the surgical area to ensure that images are acquired from multiple perspectives including front, back, left, right and top, so as to fully capture visual information within the surgical area;

[0014] Synchronous control unit: used to control the synchronous operation of all high-resolution cameras and depth sensors. Specifically, the synchronous acquisition of data is ensured through timing control. The time synchronization protocol is adopted to ensure that the internal clocks of all devices are synchronized within the error range of microsecond level. Each device will adjust the acquisition time point according to the following formula , the formula for the whole acquisition time point is:

[0015] ,in, is the originally planned data collection time, is the time correction value received from the time server;

[0016] Data fusion unit: used to fuse data from cameras at different angles and depth sensors, specifically using stereo vision matching and deep learning models to analyze and synthesize the captured images and depth information to generate an accurate three-dimensional view of the surgical area and its objects;

[0017] Optimization processing unit: optimizes the fused data to eliminate visual distortion or depth inconsistency.

[0018] Furthermore, the data fusion unit includes:

[0019] Stereoscopic vision matching: The data fusion unit uses stereoscopic vision matching technology to analyze the images captured from the multi-angle imaging module. Based on the image pairs captured by each pair of cameras, the depth information and parallax are calculated by finding the displacement of the same feature points. Calculated by the following formula: ,in, and They are the horizontal coordinates of the same feature points in the left and right camera images respectively; then the actual depth of each point is calculated using the triangulation principle , the formula is: ,in, is the focal length of the camera, is the baseline distance between cameras;

[0020] Application of deep learning model: The data fusion unit also applies deep learning model to enhance the accuracy and completeness of image depth information. Specifically, convolutional neural network is used to learn and identify high-level features of the image. The convolutional neural network model formula is expressed as: ,in, represents the weight matrix, represents the convolution operation, is the bias term, ReLU is the activation function, and the trained model predicts the depth of each pixel in the image to generate a depth map;

[0021] Data synthesis: Finally, the weighted average method is used to synthesize the depth data generated by stereo vision matching and the convolutional neural network model to form a complete three-dimensional view.

[0022] Furthermore, the optimization processing unit includes:

[0023] Distortion correction: The optimization processing unit first performs distortion correction on the image captured by the multi-angle imaging module. Based on the radial and tangential distortion caused by the camera lens, the following distortion correction formula will be used for correction; ,

[0024] ,

[0025] in, and are the image coordinates after distortion correction; and These are the original image coordinates captured directly from the camera before distortion correction; is the radial distance from the image midpoint to the image center; is the distortion coefficient;

[0026] Depth consistency correction: Next, the optimization processing unit performs consistency correction on the depth data to identify and correct depth inconsistencies caused by perspective differences or sensor errors by comparing depth data of the same area captured from different angles and applying a weighted average method to adjust the depth value. The formula is:

[0027] ,in, is the final adjusted depth value; For the The depth value measured by the camera at each angle; Based on The weights assigned to the reliability of each angle;

[0028] High-frequency detail enhancement: Finally, a sharpening algorithm is applied to enhance image edges and textures. The specific formula is:

[0029] ,in, is the enhanced image; is the original image; is the image after applying Gaussian blur; It is the enhancement coefficient, which is used to adjust the degree of high-frequency detail enhancement.

[0030] Furthermore, the micro-dynamic detection module includes a sensor configuration unit, a signal processing unit and a data analysis unit; wherein:

[0031] Sensor configuration unit: deploys multiple high-sensitivity vibration sensors in the surgical area, including the operating table and surgical tools. The vibration sensors are used to capture tiny dynamic changes caused by physiological and mechanical factors and detect tiny vibrations with a frequency range from a few hertz to several thousand hertz.

[0032] Signal processing unit: Receives signals from the vibration sensor and performs preliminary filtering to exclude environmental noise and non-surgery-related vibrations. Specifically, a digital bandpass filter is used to retain the vibration frequency band related to the surgical process. The filtering formula is: ,in, This is the filtered output signal, indicating the time The signal value after bandpass filtering at each moment; is the input signal, representing the original data collected from the vibration sensor; and The two parameters are the center frequency and bandwidth of the bandpass filter;

[0033] Data analysis unit: conducts in-depth analysis of the filtered signal and uses fast Fourier transform to convert the signal from the time domain to the frequency domain. The specific formula is: ,in, To represent the signal in the frequency domain; is the original signal value in the time domain, is the sampling point index; is the number of sample points.

[0034] Furthermore, the data processing and analysis module includes a feature extraction unit, a dynamic data analysis unit and a behavior pattern recognition unit; wherein:

[0035] Feature extraction unit: uses a preset image processing algorithm, including edge detection and feature point recognition, to extract visual features from the fused three-dimensional view output by the data fusion unit. The visual features include landmarks, boundaries, and structural information of the surgical object. The extracted visual features are the basis for locating and tracking the surgical object.

[0036] Dynamic data analysis unit: Combined with the data provided by the micro-dynamic detection module, the machine learning algorithm of the support vector machine is used to analyze the movement behavior and dynamic pattern of the surgical object, and identify the object's movement trajectory, speed and behavior pattern, including periodic movement or sudden movement;

[0037] Behavioral pattern recognition unit: Based on the data obtained by the feature extraction unit and the dynamic data analysis unit, the pattern recognition algorithm is applied to identify and classify the specific behaviors of the surgical object, and provide real-time decision support to the surgical team.

[0038] Furthermore, the dynamic data analysis unit includes:

[0039] Data preprocessing: First, receive the data from the micro-dynamic detection module, including frequency, amplitude and timestamp, and normalize the data to prepare for machine learning analysis. Normalization is to adjust the data to a uniform scale to reduce deviation and noise in the data;

[0040] Feature extraction: Extract useful features from the preprocessed data, including the peak value of the vibration frequency, the average energy, and the vibration mode within the time window;

[0041] Support vector machine classifier: Support vector machine is used for pattern recognition and classification. Support vector machine is used to distinguish different types of surgical object movement behaviors and is optimized based on the following calculation formula: ;

[0042] ,in, is the normal vector of the decision surface, is the bias term, is the slack variable, is the regularization parameter, is the eigenvector, is the class label;

[0043] Dynamic pattern recognition: Based on the output of SVM, the specific dynamic patterns of surgical objects are identified and classified, including steady motion, sudden rapid motion, or periodic motion.

[0044] Furthermore, the behavior pattern recognition unit includes:

[0045] Data integration: First, the data is received from feature extraction and dynamic data analysis. The feature extraction unit provides the visual features of the surgical object, including the location of the boundaries and landmarks, while the dynamic data analysis provides the motion features of the object, including speed and motion trajectory. The received data is integrated into a comprehensive feature vector for subsequent pattern recognition.

[0046] Pattern recognition algorithm selection: Select an appropriate pattern recognition algorithm to process the integrated feature vector. Appropriate pattern recognition algorithms include decision trees, random forests, or neural networks.

[0047] Training and classification: Use the selected pattern recognition algorithm to train and classify the behavior patterns of surgical subjects;

[0048] Behavior pattern output unit: The trained pattern recognition model classifies the input feature vector into one or more behavior patterns, including stable movement, sudden rapid movement, or periodic movement, which is used to assist surgical decision-making and improve surgical safety.

[0049] Furthermore, the dynamic calibration synchronization module includes a data receiving unit, a view updating unit, a synchronization control unit and a calibration verification unit; wherein:

[0050] Data receiving unit: used to receive information about the precise position, motion trajectory and behavior pattern of the surgical object from the data processing and analysis module, including the three-dimensional coordinates of the object and the predicted movement path, so as to ensure that the synchronization module has a real-time updated state of the surgical object;

[0051] View update unit: Based on the received data, the view update is calculated. Specifically, the actual position of the surgical object is mapped to the view space of the AR device using a transformation matrix. The calculation formula of the transformation matrix is ​​as follows:

[0052] ,in, is the three-dimensional coordinate vector of the surgical object in the actual surgical space, is the transformation matrix from real space to AR device view space, is the mapped view space coordinate;

[0053] Synchronization control unit: used to control the real-time update of the view of the AR device to ensure the synchronization and accuracy of the view. Specifically, a high-frequency update strategy is used to adapt to the dynamic changes of the surgical object and ensure that the position of the surgical object in the AR view is completely synchronized with the position in the actual surgery. The synchronization algorithm takes into account the delay of data transmission and processing, adjusts the timing of view updates, and ensures that visual lag is minimized;

[0054] Calibration verification unit: used to verify the accuracy of calibration. By comparing with the markers placed in the surgical area, the deviation between the display of the AR view and the actual position is detected. When the deviation exceeds the preset threshold, it will automatically recalibrate and adjust the transformation matrix To reduce errors.

[0055] Furthermore, the enhanced visual mapping module includes a mapping data receiving unit, a visual overlay processing unit, a visual optimization unit and a feedback adjustment unit; wherein:

[0056] Mapping data receiving unit: receiving data results from the data processing and analysis module and the dynamic calibration synchronization module, including the precise position and motion trajectory of the surgical object, as well as visual information of the surgery;

[0057] Visual overlay processing unit: uses graphics processing technology to convert received data into a format suitable for display on AR devices, synthesizes 3D data with 2D images through software algorithms, and creates visual overlay effects that can be displayed on AR glasses or screens, including converting data into a graphics layer and then synthesizing it with a real-time video or image data layer;

[0058] Visual optimization unit: used to adjust the visual quality of the overlay view, including brightness, contrast, and color. This is done through a dynamic adjustment algorithm to ensure that the overlay image is clearly visible under different ambient light conditions. The adjustment algorithm formula is: ,in, is the original image data, and They are coefficients for adjusting brightness and contrast respectively; the visual optimization unit also includes a color correction algorithm for adjusting the image color to adapt to the lighting conditions in the operating room and enhance the naturalness and recognition of the visual effect;

[0059] Feedback adjustment unit: used to adjust the visual parameters of the mapping content in real time based on the feedback from medical staff and the data provided by environmental sensors.

[0060] Beneficial effects of the present invention:

[0061] The present invention, by integrating advanced dynamic data analysis and calibration synchronization technology, can accurately track and display the position of the surgical object and its dynamic changes in real time, which not only greatly improves the accuracy of surgical navigation, but also reduces errors and risks during the operation through precise visual assistance, thereby significantly improving the safety of the operation.

[0062] The present invention, by enhancing the visual mapping module, can automatically adjust the brightness, contrast and color of the superimposed image according to different surgical environments, which ensures that medical staff can obtain clear and accurate visual information under various lighting conditions, further optimizes the visual experience during the operation, and reduces surgical delays or errors caused by poor visual information.

[0063] The present invention speeds up the surgical process and reduces the operation time through real-time dynamic tracking and efficient processing of visual information. The high responsiveness and accuracy of the system enable the surgical team to make decisions quickly and effectively deal with various complex situations that arise during the operation. This not only improves the surgical efficiency, but also increases the smoothness of the surgical process, thereby improving the patient's overall treatment experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of a high-precision tracking and positioning system for dynamic objects according to an embodiment of the present invention;

[0066] Figure 2 Schematic diagram of a data processing and analysis module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0068] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. The words "include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0069] like Figure 1-Figure 2 As shown, a high-precision tracking and positioning system for dynamic objects based on AR medical navigation includes a multi-angle imaging module, a micro-dynamic detection module, a data processing and analysis module, a dynamic calibration and synchronization module, and an enhanced visual mapping module; wherein,

[0070] Multi-angle imaging module: equipped with high-resolution cameras and depth sensors at multiple angles to capture detailed visual and depth information within the surgical area to provide a three-dimensional view of the surgical object and its surroundings;

[0071] Micro-dynamic detection module: integrated with high-sensitivity vibration sensor to capture and record small dynamic changes of surgical objects caused by physiological or mechanical factors, so as to enhance the tracking details and accuracy;

[0072] Data processing and analysis module: Based on the data from the multi-angle imaging module and the micro-dynamic detection module, the preset image processing algorithm and machine learning technology are used for analysis to extract the position, motion trajectory and behavior pattern of the object, and provide real-time position and predicted trajectory data for the dynamic calibration synchronization module;

[0073] Dynamic calibration and synchronization module: adjusts the view displayed by the AR device according to the information provided by the data processing and analysis module to ensure that the position of the surgical object in the AR view is completely synchronized with the position in the actual surgery;

[0074] Enhanced visual mapping module: presents the data results of the data processing and analysis module and the dynamic calibration and synchronization module in the form of visual overlay on the medical staff's AR device. During the process, the brightness, contrast and color of the mapping content are adjusted through visual optimization technology to adapt to different operating environments and lighting conditions, providing clear and accurate visual guidance.

[0075] The multi-angle imaging module includes a configuration unit, a synchronization control unit, a data fusion unit and an optimization processing unit; wherein:

[0076] Configuration unit: equipped with high-resolution cameras at at least three angles and at least two depth sensors, the cameras and sensors are evenly spaced around the surgical area to ensure that images are acquired from multiple perspectives including front, back, left, right and top, so as to fully capture visual information within the surgical area;

[0077] Synchronous control unit: used to control the synchronous operation of all high-resolution cameras and depth sensors. Specifically, it ensures the synchronous acquisition of data through timing control. It uses time synchronization protocols (such as Network Time Protocol, NTP) to ensure that the internal clocks of all devices are synchronized within the error range of microseconds. Each device will adjust the acquisition time point according to the following formula , the formula for the whole acquisition time point is:

[0078] ,in, is the originally planned data collection time, is the time correction value received from the time server. This approach ensures the temporal consistency of data obtained from different devices, allowing accurate reconstruction and analysis of the three-dimensional view of the surgical scene;

[0079] Data fusion unit: used to fuse data from cameras at different angles and depth sensors, specifically using stereo vision matching and deep learning models to analyze and synthesize the captured images and depth information to generate an accurate three-dimensional view of the surgical area and its objects;

[0080] Optimization processing unit: Optimizes the fused data to eliminate visual distortion or depth inconsistency, ensuring that the generated 3D view is highly accurate and reliable in terms of geometry and vision.

[0081] The data fusion unit includes:

[0082] Stereoscopic vision matching: The data fusion unit uses stereoscopic vision matching technology to analyze the images captured from the multi-angle imaging module. Based on the image pairs captured by each pair of cameras, the depth information is calculated by finding the displacement of the same feature points (called parallax). Calculated by the following formula: ,in, and They are the horizontal coordinates of the same feature points in the left and right camera images respectively; then the actual depth of each point is calculated using the triangulation principle , the formula is: ,in, is the focal length of the camera, is the baseline distance between cameras;

[0083] Application of deep learning model: The data fusion unit also applies deep learning model to enhance the accuracy and completeness of image depth information. Specifically, convolutional neural network (CNN) is used to learn and identify high-level features of images. The convolutional neural network model formula is expressed as: ,in, represents the weight matrix, represents the convolution operation, is the bias term, ReLU is the activation function, and the trained model predicts the depth of each pixel in the image to generate a depth map;

[0084] Data synthesis: Finally, the weighted averaging method is used to synthesize the depth data generated by the stereo vision matching and the convolutional neural network model to form a complete three-dimensional view to ensure the continuity and accuracy of the three-dimensional view. By combining the above-mentioned method of stereo vision matching technology and deep learning model, the data fusion unit can effectively improve the accuracy and detail of the three-dimensional view of the surgical area and its objects, which not only improves the accuracy of surgical navigation, but also enhances surgical safety, because medical staff can obtain clearer and more accurate views to guide the surgical process. In addition, through automated and optimized data processing, the need for manual adjustment is greatly reduced, thereby improving surgical efficiency.

[0085] The optimization processing unit includes:

[0086] Distortion correction: The optimization processing unit first performs distortion correction on the image captured by the multi-angle imaging module. Based on the radial and tangential distortion caused by the camera lens, the following distortion correction formula will be used for correction; ,

[0087] ,

[0088] in, and are the image coordinates after distortion correction, i.e., the coordinates in the image captured by the camera after correction processing to eliminate the distortion effect; and These are the original image coordinates captured directly from the camera before distortion correction; The radial distance from the image midpoint to the image center is the straight-line distance from the image center to a specific point, usually in pixels; Distortion coefficients are obtained during camera calibration. They describe the degree of radial distortion in the image and are usually calculated through experiments or preset calibration procedures.

[0089] Depth consistency correction: Next, the optimization processing unit performs consistency correction on the depth data to identify and correct depth inconsistencies caused by perspective differences or sensor errors by comparing depth data of the same area captured from different angles and applying a weighted average method to adjust the depth value. The formula is: ,in, is the final adjusted depth value, which is the result of consistency correction of depth data obtained from multiple cameras; For the The depth value measured by the camera at different angles. In a surgical environment, cameras at different angles may measure different depth values ​​due to differences in viewing angles and sensor characteristics. Based on The weights assigned to the reliability of each angle reflect the credibility of each camera's perspective and are usually assigned based on factors such as the clarity of the perspective, the distance and angle between the camera and the surgical object, etc.

[0090] High-frequency detail enhancement: Finally, a sharpening algorithm is applied to enhance image edges and textures. The specific formula is:

[0091] ,in, The enhanced image is the result of high-frequency detail processing on the original image, which is used to improve the details and edge clarity in the image; The original image is the unprocessed image data directly obtained from the camera or image acquisition device; The image after applying Gaussian blur is used to remove high-frequency noise from the original image, and then combined with the original image to enhance details; is the enhancement coefficient, which is used to adjust the degree of high-frequency detail enhancement. This parameter determines the degree of emphasis on the difference between the original image and the blurred image, thereby controlling the strength of detail enhancement;

[0092] The application of the above-mentioned optimization processing unit steps can significantly improve the visual quality and depth consistency of the generated three-dimensional views. By reducing distortion and enhancing depth consistency, the system can provide more accurate and realistic views, which helps medical staff make better decisions in complex operations. In addition, the enhancement of high-frequency details makes the subtle structures of the surgical area clearer, further improving the accuracy and safety of the operation.

[0093] The micro-dynamic detection module includes a sensor configuration unit, a signal processing unit and a data analysis unit; wherein:

[0094] Sensor configuration unit: deploys multiple high-sensitivity vibration sensors in the surgical area, including the operating table and surgical tools. The vibration sensors are used to capture tiny dynamic changes caused by physiological factors (such as heartbeat and breathing) and mechanical factors (such as the operation of surgical tools), and are used to detect tiny vibrations with a frequency range from a few hertz to several thousand hertz;

[0095] Signal processing unit: Receives signals from the vibration sensor and performs preliminary filtering to exclude environmental noise and non-surgery-related vibrations. Specifically, a digital bandpass filter is used to retain the vibration frequency band related to the surgical process. The filtering formula is: ,in, This is the filtered output signal, indicating the time The signal value after bandpass filtering at all times. This output is used in the subsequent data analysis unit to help focus on the vibration frequency related to the surgery; is the input signal, which represents the raw data collected from the vibration sensor, including vibrations generated by various physiological and mechanical activities in the surgical area; and These two parameters are the center frequency and bandwidth of the bandpass filter. The center frequency is specifies the primary frequency of interest for the filter, while the bandwidth Determines the frequency range around the center frequency, that is, the signal frequency range allowed to pass through the filter; This is the formula for a bandpass filter, based on a Gaussian function modified to emphasize the center frequency nearby signals while suppressing those far away frequency components, thus ensuring that only vibration data directly related to the surgical procedure is analyzed;

[0096] Data analysis unit: Perform in-depth analysis on the filtered signal and use fast Fourier transform (FFT) to convert the signal from time domain to frequency domain. The specific formula is: ,in, To represent the signal in the frequency domain, It is a specific frequency point. This signal is obtained after applying the fast Fourier transform (FFT) and is used to represent the distribution of the original time signal at each frequency. is the original signal value in the time domain, is the sampling point index, which is the filtered signal passed from the signal processing unit for FFT analysis; The number of sample points indicates the total number of data points considered in the FFT analysis. This parameter directly affects the frequency resolution and the level of detail of the FFT operation. is the calculation formula of fast Fourier transform, which is used to convert the time domain signal to the frequency domain. is a complex exponential function, where is the imaginary unit, which represents the frequency At each time point Fast Fourier transform analysis can help identify various frequency components. Through the characteristics of these frequency components (such as amplitude and phase), the dynamic behavior of the surgical object, such as periodic motion or sudden vibration, can be accurately identified.

[0097] The above-mentioned micro-dynamic detection module can accurately capture tiny dynamic changes of the surgical object by integrating high-sensitivity vibration sensors and advanced signal processing technology. This technology provides detailed information about the real-time dynamics of the surgical object, increasing the accuracy and safety during the operation. By accurately monitoring the dynamic characteristics of the surgical object, medical staff can obtain more valuable information to optimize surgical strategies and improve the success rate of the operation. In addition, it reduces uncertainty and risks during the operation and improves the overall efficiency of the operation and patient safety.

[0098] The data processing and analysis module includes a feature extraction unit, a dynamic data analysis unit and a behavior pattern recognition unit; wherein:

[0099] Feature extraction unit: uses a preset image processing algorithm, including edge detection and feature point recognition, to extract visual features from the fused three-dimensional view output by the data fusion unit. The visual features include landmarks, boundaries, and structural information of the surgical object. The extracted visual features are the basis for locating and tracking the surgical object.

[0100] Dynamic data analysis unit: Combined with the data provided by the micro-dynamic detection module, the machine learning algorithm of the support vector machine is used to analyze the movement behavior and dynamic pattern of the surgical object, and identify the movement trajectory, speed and behavior pattern of the object, including periodic movement or sudden movement;

[0101] Behavior pattern recognition unit: Based on the data obtained by the feature extraction unit and the dynamic data analysis unit, the pattern recognition algorithm is used to identify and classify the specific behavior of the surgical object. The analysis of this behavior pattern recognition unit is used to assist in predicting various situations during the operation and provide real-time decision support to the surgical team; by integrating and analyzing data from different modules, the data processing and analysis module can accurately provide dynamic information of the surgical object, which enhances the accuracy and practicality of the surgical navigation system and improves the safety and success rate of the operation. Real-time behavior pattern recognition enables the medical team to effectively respond to various situations during the operation, optimize surgical strategies, and reduce surgical risks.

[0102] The dynamic data analysis unit includes:

[0103] Data preprocessing: First, receive the data from the micro-dynamic detection module, including frequency, amplitude and timestamp, and normalize the data to prepare for machine learning analysis. Normalization is to adjust the data to a uniform scale to reduce deviation and noise in the data;

[0104] Feature extraction: Extract useful features from the preprocessed data, including the peak value of the vibration frequency, the average energy, and the vibration pattern within the time window. These features are crucial for describing the dynamic state of the surgical object because they reflect the specific behavior of the object during the operation.

[0105] Support vector machine classifier: Use support vector machine for pattern recognition and classification. Support vector machine is a supervised learning algorithm used to identify patterns in data and classify them. In this application, support vector machine is used to distinguish different types of surgical object movement behaviors and is optimized based on the following calculation formula:

[0106] ;

[0107] ,in, is the normal vector of the decision surface, is the bias term, is the slack variable, is the regularization parameter, is the eigenvector, is the class label, which is classified by constructing a hyperplane that maximizes the margin between classes, which helps to accurately identify the movement trajectory and behavior pattern of the surgical object;

[0108] Dynamic pattern recognition: Based on the output of SVM, the specific dynamic patterns of surgical objects are identified and classified, including stable movements, sudden rapid movements, or periodic movements. With this information, complex situations that may arise during surgery can be predicted, providing more accurate decision support for medical staff.

[0109] The behavioral pattern recognition unit includes:

[0110] Data integration: First, the data is received from feature extraction and dynamic data analysis. The feature extraction unit provides the visual features of the surgical object, including the location of the boundaries and landmarks, while the dynamic data analysis provides the motion features of the object, including speed and motion trajectory. The received data is integrated into a comprehensive feature vector for subsequent pattern recognition.

[0111] Pattern recognition algorithm selection: Select a suitable pattern recognition algorithm to process the integrated feature vector. Suitable pattern recognition algorithms include decision trees, random forests, or neural networks. Each algorithm has its advantages. For example, decision trees are suitable for explaining simple scenarios, while neural networks are more suitable for processing complex pattern recognition tasks.

[0112] Training and classification: Use the selected pattern recognition algorithm to train and classify the behavior patterns of surgical subjects;

[0113] Behavior pattern output unit: The trained pattern recognition model classifies the input feature vector into one or more behavior patterns, including stable movement, sudden rapid movement or periodic movement. The behavior pattern information will be output to other parts of the system to assist surgical decision-making and improve surgical safety.

[0114] The behavioral pattern recognition unit can accurately identify and classify the behavior of surgical subjects during surgery by applying advanced pattern recognition technology. This not only improves the accuracy and safety of the surgery, but also enables the medical team to respond to changes in the surgery in a timely manner, optimize surgical strategies, and reduce surgical risks. In addition, the application of this technology can greatly improve the monitoring efficiency and response speed of the surgical process, and improve the overall surgical quality and patient satisfaction.

[0115] The dynamic calibration synchronization module includes a data receiving unit, a view updating unit, a synchronization control unit and a calibration verification unit; wherein:

[0116] Data receiving unit: used to receive information about the precise position, motion trajectory and behavior pattern of the surgical object from the data processing and analysis module, including the three-dimensional coordinates of the object and the predicted movement path, so as to ensure that the synchronization module has a real-time updated state of the surgical object;

[0117] View update unit: Based on the received data, the view update is calculated. Specifically, the actual position of the surgical object is mapped to the view space of the AR device using a transformation matrix. The calculation formula of the transformation matrix is ​​as follows:

[0118] ,in, is the three-dimensional coordinate vector of the surgical object in the actual surgical space, is the transformation matrix from real space to AR device view space, is the mapped view space coordinate;

[0119] Synchronization control unit: used to control the real-time update of the view of the AR device to ensure the synchronization and accuracy of the view. Specifically, a high-frequency update strategy is used to adapt to the dynamic changes of the surgical object and ensure that the position of the surgical object in the AR view is completely synchronized with the position in the actual surgery. The synchronization algorithm takes into account the delay of data transmission and processing, adjusts the timing of view updates, and ensures that visual lag is minimized;

[0120] Calibration verification unit: used to verify the accuracy of calibration. By comparing with the markers placed in the surgical area, the deviation between the display of the AR view and the actual position is detected. When the deviation exceeds the preset threshold, it will automatically recalibrate and adjust the transformation matrix To reduce errors;

[0121] The above-mentioned dynamic calibration and synchronization module ensures that the position of the surgical object in the AR view is completely synchronized with the position in the actual operation by updating and accurately calibrating the AR view in real time. This not only improves the accuracy of surgical navigation, but also enhances surgical safety and efficiency. Through accurate visual synchronization, doctors can better understand the surgical scene and make more accurate surgical decisions, thereby improving the success rate of surgery and patient satisfaction.

[0122] The enhanced visual mapping module includes a mapping data receiving unit, a visual overlay processing unit, a visual optimization unit, and a feedback adjustment unit; wherein:

[0123] Mapping data receiving unit: Receives data results from the data processing and analysis module and the dynamic calibration synchronization module, including the precise position and motion trajectory of the surgical object, as well as the visual information of the surgery. This information is the basis of the mapping process and ensures that the displayed content is the latest and most accurate;

[0124] Visual overlay processing unit: uses graphics processing technology to convert received data into a format suitable for display on AR devices, synthesizes 3D data with 2D images through software algorithms, and creates visual overlay effects that can be displayed on AR glasses or screens, including converting data into a graphics layer and then synthesizing it with a real-time video or image data layer;

[0125] Visual optimization unit: used to adjust the visual quality of the overlay view, including brightness, contrast, and color. This is done through a dynamic adjustment algorithm to ensure that the overlay image is clearly visible under different ambient light conditions. The adjustment algorithm formula is: ,in, is the original image data, and They are coefficients for adjusting brightness and contrast respectively; the visual optimization unit also includes a color correction algorithm for adjusting the image color to adapt to the lighting conditions in the operating room and enhance the naturalness and recognition of the visual effect;

[0126] Feedback adjustment unit: used to adjust the visual parameters of the mapping content in real time based on the feedback from medical staff and the data provided by environmental sensors (such as light sensors), ensuring that the visual overlay always maintains the best quality during the operation and that medical staff can operate under the best visual conditions;

[0127] The above-mentioned enhanced visual mapping module significantly improves the usability and effectiveness of surgical navigation by accurately presenting surgery-related data on AR devices in the form of visual overlays and adjusting the display effect through advanced visual optimization technology. By optimizing brightness, contrast and color, it ensures that medical staff can clearly see key information in the surgical area under any lighting conditions, thereby improving surgical accuracy and safety. In addition, the real-time feedback adjustment function allows the system to automatically optimize display settings based on actual usage, enhancing user experience and surgical efficiency.

[0128] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-precision tracking and positioning system for dynamic objects based on AR medical navigation, characterized in that: It includes a multi-angle imaging module, a micro-dynamic detection module, a data processing and analysis module, a dynamic calibration and synchronization module, and an enhanced visual mapping module; among which, Multi-angle imaging module: A high-resolution camera and depth sensor configured at multiple angles are used to capture detailed visual and depth information within the surgical area to provide a three-dimensional view of the surgical object and its surroundings. The multi-angle imaging module includes a configuration unit, a synchronization control unit, a data fusion unit, and an optimization processing unit; wherein: Configuration unit: equipped with high-resolution cameras at at least three angles and at least two depth sensors, the cameras and sensors are evenly spaced around the surgical area to ensure that images are acquired from multiple perspectives including front, back, left, right and top, so as to fully capture visual information within the surgical area; Synchronous control unit: used to control the synchronous operation of all high-resolution cameras and depth sensors. Specifically, the synchronous acquisition of data is ensured through timing control. The time synchronization protocol is adopted to ensure that the internal clocks of all devices are synchronized within the error range of microsecond level. Each device will adjust the acquisition time point according to the following formula , the formula for the whole acquisition time point is: ,in, is the originally planned data collection time, is the time correction value received from the time server; Data fusion unit: used to fuse data from cameras at different angles and depth sensors, specifically using stereo vision matching and deep learning models to analyze and synthesize the captured images and depth information to generate an accurate three-dimensional view of the surgical area and its objects; Optimization processing unit: optimizes the fused data to eliminate visual distortion or depth inconsistency; The data fusion unit comprises: Stereoscopic vision matching: The data fusion unit uses stereoscopic vision matching technology to analyze the images captured from the multi-angle imaging module. Based on the image pairs captured by each pair of cameras, the depth information and parallax are calculated by finding the displacement of the same feature points. Calculated by the following formula: ,in, and They are the horizontal coordinates of the same feature points in the left and right camera images respectively; then the actual depth of each point is calculated using the triangulation principle , the formula is: ,in, is the focal length of the camera, is the baseline distance between the cameras; Application of deep learning model: The data fusion unit also applies a deep learning model to enhance the accuracy and completeness of the image depth information. Specifically, a convolutional neural network is used to learn and identify the high-level features of the image. The convolutional neural network model formula is expressed as: ,in, represents the weight matrix, * represents the convolution operation, is the bias term, ReLU is the activation function, and the trained model predicts the depth of each pixel in the image to generate a depth map; Data synthesis: Finally, the weighted average method is used to synthesize the depth data generated by stereo vision matching and the convolutional neural network model to form a complete three-dimensional view; The optimization processing unit comprises: Distortion correction: The optimization processing unit first performs distortion correction on the image captured by the multi-angle imaging module. Based on the radial and tangential distortion caused by the camera lens, the following distortion correction formula will be used for correction; , ， in, and are the image coordinates after distortion correction; and are the original image coordinates captured directly from the camera before distortion correction; is the radial distance from the image midpoint to the image center; is the distortion coefficient; Depth consistency correction: Next, the optimization processing unit performs consistency correction on the depth data to identify and correct depth inconsistencies caused by perspective differences or sensor errors by comparing depth data of the same area captured from different angles and applying a weighted average method to adjust the depth value. The formula is: ,in, is the final adjusted depth value; For the The depth value measured by the camera at each angle; Based on The weights assigned to the reliability of each angle; High-frequency detail enhancement: Finally, a sharpening algorithm is applied to enhance image edges and textures. The specific formula is: ,in, is the enhanced image; is the original image; is the image after applying Gaussian blur; is the enhancement coefficient, which is used to adjust the degree of high-frequency detail enhancement; Micro-dynamic detection module: integrated with high-sensitivity vibration sensor to capture and record small dynamic changes of surgical objects caused by physiological or mechanical factors, so as to enhance the tracking details and accuracy; Data processing and analysis module: Based on the data from the multi-angle imaging module and the micro-dynamic detection module, the preset image processing algorithm and machine learning technology are used for analysis to extract the position, motion trajectory and behavior pattern of the object; Dynamic calibration and synchronization module: adjusts the view displayed by the AR device according to the information provided by the data processing and analysis module to ensure that the position of the surgical object in the AR view is completely synchronized with the position in the actual surgery; Enhance the visual mapping module; present the data results of the data processing and analysis module and the dynamic calibration and synchronization module on the medical staff's AR device in the form of visual overlay. During the process, adjust the brightness, contrast and color of the mapping content through visual optimization technology.

2. According to claim 1, a dynamic object high-precision tracking and positioning system based on AR medical navigation is characterized in that: The micro-dynamic detection module includes a sensor configuration unit, a signal processing unit and a data analysis unit; wherein: Sensor configuration unit: deploys multiple high-sensitivity vibration sensors in the surgical area, including the operating table and surgical tools. The vibration sensors are used to capture tiny dynamic changes caused by physiological and mechanical factors and detect tiny vibrations with a frequency range from a few hertz to several thousand hertz. Signal processing unit: Receives the signal from the vibration sensor and performs preliminary filtering to exclude environmental noise and non-surgery related vibrations. The digital bandpass filter is used for processing. The formula is: ,in, is the filtered output signal, indicating the time The signal value after bandpass filtering at each moment; is the input signal, representing the original data collected from the vibration sensor; and are the center frequency and bandwidth of the bandpass filter respectively; Data analysis unit: Perform in-depth analysis on the filtered signal and use fast Fourier transform to convert the signal from time domain to frequency domain. The specific formula is: ,in, Representing a signal in the frequency domain; is the original signal value in the time domain, is the sampling point index; is the number of sample points.

3. According to claim 1, a dynamic object high-precision tracking and positioning system based on AR medical navigation is characterized in that: The data processing and analysis module includes a feature extraction unit, a dynamic data analysis unit and a behavior pattern recognition unit; wherein: Feature extraction unit: uses a preset image processing algorithm, including edge detection and feature point recognition, to extract visual features from the fused three-dimensional view output by the data fusion unit. The visual features include landmarks, boundaries, and structural information of the surgical object. The extracted visual features are the basis for locating and tracking the surgical object. Dynamic data analysis unit: Combined with the data provided by the micro-dynamic detection module, the machine learning algorithm of the support vector machine is used to analyze the movement behavior and dynamic pattern of the surgical object, and identify the object's movement trajectory, speed and behavior pattern, including periodic movement or sudden movement; Behavioral pattern recognition unit: Based on the data obtained by the feature extraction unit and the dynamic data analysis unit, the pattern recognition algorithm is applied to identify and classify the specific behaviors of the surgical object, and provide real-time decision support to the surgical team.

4. According to claim 3, a dynamic object high-precision tracking and positioning system based on AR medical navigation is characterized in that: The dynamic data analysis unit comprises: Data preprocessing: First, receive the data from the micro-dynamic detection module, including frequency, amplitude and timestamp, and normalize the data to prepare for machine learning analysis. Normalization is to adjust the data to a uniform scale to reduce deviation and noise in the data; Feature extraction: Extract useful features from the preprocessed data, including the peak value of the vibration frequency, the average energy, and the vibration mode within the time window; Support vector machine classifier: Support vector machine is used for pattern recognition and classification. Support vector machine is used to distinguish different types of surgical object movement behaviors and is optimized based on the following calculation formula: ； ,in, is the normal vector of the decision surface, is the bias term, is the slack variable, is the regularization parameter, is the eigenvector, is the class label; Dynamic pattern recognition: Based on the output of SVM, the specific dynamic patterns of surgical objects are identified and classified, including steady motion, sudden rapid motion, or periodic motion.

5. The high-precision tracking and positioning system for dynamic objects based on AR medical navigation according to claim 4, characterized in that: The behavior pattern recognition unit comprises: Data integration: First, the data is received from feature extraction and dynamic data analysis. The feature extraction unit provides the visual features of the surgical object, including the location of the boundaries and landmarks, while the dynamic data analysis provides the motion features of the object, including speed and motion trajectory. The received data is integrated into a comprehensive feature vector for subsequent pattern recognition. Pattern recognition algorithm selection: Select an appropriate pattern recognition algorithm to process the integrated feature vector. Appropriate pattern recognition algorithms include decision trees, random forests, or neural networks. Training and classification: Use the selected pattern recognition algorithm to train and classify the behavior patterns of surgical subjects; Behavior pattern output unit: The trained pattern recognition model classifies the input feature vector into one or more behavior patterns, including stable movement, sudden rapid movement, or periodic movement, which is used to assist surgical decision-making and improve surgical safety.

6. The high-precision tracking and positioning system for dynamic objects based on AR medical navigation according to claim 1, characterized in that: The dynamic calibration synchronization module includes a data receiving unit, a view updating unit, a synchronization control unit and a calibration verification unit; wherein: Data receiving unit: used to receive information about the precise position, motion trajectory and behavior pattern of the surgical object from the data processing and analysis module, including the three-dimensional coordinates of the object and the predicted movement path, so as to ensure that the synchronization module has a real-time updated state of the surgical object; View update unit: Based on the received data, the view update is calculated. Specifically, the actual position of the surgical object is mapped to the view space of the AR device using a transformation matrix. The calculation formula of the transformation matrix is ​​as follows: ,in, is the three-dimensional coordinate vector of the surgical object in the actual surgical space, is the transformation matrix from real space to AR device view space, is the mapped view space coordinate; Synchronization control unit: used to control the real-time update of the view of the AR device to ensure the synchronization and accuracy of the view. Specifically, a high-frequency update strategy is used to adapt to the dynamic changes of the surgical object and ensure that the position of the surgical object in the AR view is completely synchronized with the position in the actual surgery. The synchronization algorithm takes into account the delay of data transmission and processing, adjusts the timing of view updates, and ensures that visual lag is minimized; Calibration verification unit: used to verify the accuracy of calibration. By comparing with the markers placed in the surgical area, the deviation between the display of the AR view and the actual position is detected. When the deviation exceeds the preset threshold, it will automatically recalibrate and adjust the transformation matrix To reduce errors.

7. The high-precision tracking and positioning system for dynamic objects based on AR medical navigation according to claim 1, characterized in that: The enhanced visual mapping module includes a mapping data receiving unit, a visual overlay processing unit, a visual optimization unit and a feedback adjustment unit; wherein: Mapping data receiving unit: receiving data results from the data processing and analysis module and the dynamic calibration synchronization module, including the precise position and motion trajectory of the surgical object, as well as visual information of the surgery; Visual overlay processing unit: uses graphics processing technology to convert received data into a format suitable for display on AR devices, synthesizes 3D data with 2D images through software algorithms, and creates visual overlay effects that can be displayed on AR glasses or screens, including converting data into a graphics layer and then synthesizing it with a real-time video or image data layer; Visual optimization unit: used to adjust the visual quality of the overlay view, including brightness, contrast, and color. This is done through a dynamic adjustment algorithm to ensure that the overlay image is clearly visible under different ambient light conditions. The adjustment algorithm formula is: ,in, is the original image data, and They are coefficients for adjusting brightness and contrast respectively; the visual optimization unit also includes a color correction algorithm for adjusting the image color to adapt to the lighting conditions in the operating room and enhance the naturalness and recognition of the visual effect; Feedback adjustment unit: used to adjust the visual parameters of the mapping content in real time based on the feedback from medical staff and the data provided by environmental sensors.

Images