Real-time operation remote video consultation system and method based on virtual reality

Abstract

The invention provides a real-time operation remote video consultation system and method based on virtual reality, and the system comprises a driving data collection module which collects multi-modal data; the data processing module is used for carrying out fusion processing on data from different acquisition sources; the virtual image driving module is used for reconstructing a virtual scene of an operation site and performing three-dimensional modeling and rendering on an image acquired by the high-definition camera and an internal structure image acquired by the medical imaging equipment; the template matching module is used for collecting and sorting standard action templates of common surgical operations and matching the standard action templates; the data collaboration module is fused with the virtual scene to display the medical data of the patient; and the 5G robot assistance module remotely controls the robot to execute an action corresponding to the virtual reality scene through 5G. According to the invention, audio and scene fusion is realized, scene fidelity is improved, and consultation immersion is enhanced. Interaction between experts and virtual scenes in consultation can be achieved, consultation efficiency and sense of participation are improved, doctors are assisted in standard operation, and surgical risks are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of medical virtual reality, and particularly to a real-time surgical remote video consultation system and method based on virtual reality. Background Art

[0002] With the continuous development of medical technology, the complexity of surgeries is increasing day by day, and the professional skills and experience requirements for the surgical team are becoming more and more stringent. However, the distribution of high-quality medical resources is uneven. When facing complex surgeries, medical institutions in remote areas often have difficulty obtaining timely and effective expert consultation support. Traditional remote consultation methods are mainly based on two-dimensional video communication and have many limitations. On the one hand, consultation experts cannot intuitively and comprehensively perceive the three-dimensional spatial information and fine operation details of the surgical site, making it difficult to make accurate judgments. On the other hand, the interaction means are single, lacking natural and efficient multimodal interaction methods, which affect the consultation efficiency and quality. In addition, existing systems have deficiencies in the integration and collaborative browsing of medical imaging data, and cannot meet the needs of comprehensive analysis of various imaging data during the surgical process. In terms of surgical operation assistance, although robotic technology has been applied in the medical field, its combination with the remote consultation system is not close enough to fully utilize the auxiliary advantages of robots in surgeries. Therefore, it is urgent to develop an innovative real-time surgical remote video consultation system based on virtual reality. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a real-time surgical remote video consultation system and method based on virtual reality to solve the deficiencies of the prior art described in the background art.

[0004] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0005] A real-time surgical remote video consultation system based on virtual reality, comprising:

[0006] A driving data acquisition module, configured to acquire visual data, audio data, surgical instrument movement tracking data, and patient vital sign data of the surgical scene;

[0007] A data processing module, which performs fusion processing on data from different acquisition sources, and calibrates, compresses data, and performs security encryption on visual data, audio data, surgical instrument movement data, and patient vital sign data in terms of time and space;

[0008] The virtual avatar driving module uses virtual reality technology to reconstruct the virtual scene of the surgical site, performs 3D modeling and rendering on the images collected by the high-definition camera and the internal structure images collected by the medical imaging equipment, constructs a realistic surgical scene model, and based on the action tracking data of the surgeon, constructs and drives the doctor avatar in the virtual scene to perform real-time action simulation and real-time action interaction with the doctor avatar;

[0009] The template matching module collects and collates a large number of standard action templates for common surgical operations, establishes an action template library, and matches the real-time actions of the surgeon with the templates in the action template library;

[0010] The data collaboration module conducts data docking with the hospital's Picture Archiving and Communication System (PACS) and Hospital Information System (HIS), and integrates with the virtual scene to display the patient's medical data;

[0011] The 5G robot assistance module, when conducting practical operations in the virtual reality scene, remotely controls the robot through 5G to execute actions corresponding to the virtual reality scene.

[0012] Optionally, the data processing module performs fusion processing on data from different acquisition sources, calibrates visual data, audio data, surgical instrument movement data, and patient vital sign data in terms of time and space. Specifically:

[0013] Let be the i-th eigenvalue of the visual data at time t, be the j-th eigenvalue of the audio data at time t, be the k-th eigenvalue of the surgical instrument movement data at time t, be the L-th eigenvalue of the patient vital sign data at time t. By constructing a multi-layer convolutional neural network, the input of the network is the sequence data within a certain time window, and the output is the synchronized multi-modal data The training objective of the network is to minimize the synchronization error where is the target data after real synchronization. The convolutional layer of the CNN extracts features from the input data through the filter The formula is f is the activation function, is the input data, b c is the bias term. The pooling layer performs downsampling operations, and the fully connected layer performs data fusion and mapping to finally obtain the synchronized output;

[0014] The spatial calibration algorithm uses a deep learning object detection algorithm to detect surgical instruments and patient body parts in the visual data, and obtains their coordinates in the image Meanwhile, the true coordinates of the instrument in the three-dimensional space are obtained based on the surgical instrument motion tracking data. Through the pre-established camera calibration matrix K and coordinate transformation matrix T, the coordinates in the visual data are converted into spatial coordinates consistent with the motion tracking data. The formula is Then, the spatial calibration error is calculated. The coordinate transformation matrix T is adjusted by the gradient descent method to minimize the error and achieve spatial calibration.

[0015] Optionally, the data processing module performs fusion processing on data from different acquisition sources, and compresses the visual data, audio data, surgical instrument motion data, and patient vital sign data in terms of time and space. Specifically:

[0016] Image data compression is performed based on the GAN-based image compression model: Let G be the generator network and D be the discriminator network. For the original high-definition surgical image x, the generator network G attempts to generate a compressed image G(z), where z is a random noise vector. The task of the discriminator network D is to distinguish the real image x from the generated compressed image G(z).

[0017] The loss function of the generator network is The loss function of the discriminator network is p x (x) is the distribution of the real image, and p z (z) is the distribution of the noise vector. Through adversarial training, the generator network learns to generate a compressed image that is similar to the original image but has a smaller data volume. In the generator network, convolutional layers and deconvolutional layers are used for image encoding and decoding operations. The encoding process formula is The decoding process formula is Where is the convolutional kernel weight, b e 、b d is the bias term, and f is the activation function.

[0018] Audio compression is performed based on the RNN: Let x(t) be the value of the audio signal at time t, h(t) be the hidden state of the RNN at time t, and y(t) be the compressed audio signal. The update formula of the RNN is h(t) = tanh(W hh h(t - 1) + W hx x(t) + b h ), where W hh is the weight matrix from the hidden layer to the hidden layer, W hx is the weight matrix from the input to the hidden layer, b h is the hidden layer bias term, and the output layer formula is y(t) = W hy h(t) + b y , where W hyis the weight matrix from the hidden layer to the output layer, and b y is the bias term of the output layer; by training the RNN, the temporal features of the audio signal are learned, redundant information is removed, and audio data compression is achieved. The training objective is to minimize the reconstruction error is the original audio signal reconstructed from the compressed audio signal.

[0019] Optionally, the data processing module performs fusion processing on data from different acquisition sources, and securely encrypts visual data, audio data, surgical instrument movement data, and patient vital sign data in terms of time and space. Specifically:

[0020] Let the feature vector sequence of the multimodal data be X = {x1, x2, ···, x T}, where x t represents the feature vector at time t, and T is the sequence length; for time t, the update formula for the hidden layer state h t of the RNN is:

[0021] h t = tanh(W hh h t-1 + W hx x t + b h )

[0022] where W hh is the weight matrix from the hidden layer to the hidden layer, which determines the influence degree of the hidden layer state at the previous moment on the hidden layer state at the current moment; W hx is the weight matrix from the input to the hidden layer, which is used to map the input feature vector to the hidden layer space; b h is the bias term of the hidden layer; tanh is the hyperbolic tangent activation function, which is used to introduce non-linear transformation so that the RNN can learn complex feature relationships;

[0023] The encryption key K is generated from the hidden layer state of the RNN. The specific formula is:

[0024] K = W hk h T + b k

[0025] where W hk is the weight matrix from the hidden layer to the key generation layer, which is used to map the hidden layer state at the final moment to the encryption key: b k is the bias term of the key generation layer; by using the RNN to process the multimodal data feature vector sequence, the information in the sequence is encoded into the hidden layer state, and finally the encryption key is generated;

[0026] Among them, during model training, with the randomness and security of the key as the objective function, the key quality is evaluated by calculating the entropy value H(K) of the key and the correlation Corr(K, X) with the data. The objective function J can be expressed as:

[0027] J = αH(K) + (1 - α)(1 - Corr(K, X))

[0028] Among them, α is a trade-off coefficient used to balance the importance of the entropy value and the correlation in the objective function.

[0029] Optionally, the virtual avatar driving module reconstructs the virtual scene of the surgical site using virtual reality technology, specifically based on game development engines such as Unity or Unreal Engine.

[0030] Optionally, the virtual avatar driving module constructs and drives the doctor avatar in the virtual scene for real-time motion simulation based on the motion tracking data of the surgeon, specifically using motion capture technology and inverse kinematics (IK) algorithms to achieve the motion simulation of the virtual doctor avatar:

[0031] Let the target position of the end effector in three-dimensional space be P t , and the current angle of joint i in the bone model be α i , and the joint angle variable vector be α = {α1, α2, ···, α n},where n is the number of joints. The IK algorithm iteratively solves the optimization problem to make the calculated position P(α) of the end effector close to the target position P t , and the optimization objective function is E(α) = ||P(α) - P t || 2 ;

[0032] During the solution process, the Jacobian matrix J(α) is used for iterative calculation. The element J ij (α) represents the sensitivity of the speed of the end effector in the i direction to the change in the angle of joint j. According to the Newton-Raphson iterative method, the update formula for the joint angle is Δ α = J(α) + ΔP, where J(α) + is the pseudo-inverse of the Jacobian matrix, and ΔP = P t - P(α) is the position error of the end effector. Through multiple iterative calculations, the joint target angle α t that meets the accuracy requirements is obtained;

[0033] According to the solved joint target angle α t , the animation interpolation algorithm is used to generate a smooth transition animation of the virtual doctor avatar from the current pose to the target pose. Let the joint angle of the current pose be α c, the transition time is T tr , for each joint i, the angular change θ i (t), where t ∈ [0, T tr is calculated by the interpolation formula:

[0034]

[0035] Apply the generated animation data to the skeletal model of the virtual doctor image to drive the virtual doctor image to simulate the actions of the surgical doctor in real time in the virtual reality scene.

[0036] Optionally, match with the templates in the action template library, and use the action recognition algorithm of the dynamic time warping DTW algorithm to match the real-time actions of the surgical doctor with the action templates. Specifically:

[0037] Calculate the distance matrix D between the real-time action sequence R and each action template sequence T. The element d ij of the distance matrix represents the distance metric between the feature vectors of the real-time action at time i and the template action at time j. The formula is

[0038] where d is the dimension of the action feature vector;

[0039] Let the path P = {(i1, j1), (i2, j2), ··· (i l , j l )}, where (i k , j k ) represents the kth node in the path, and l is the path length. The DTW algorithm calculates the cumulative distance of the path through the idea of dynamic programming. The recurrence formula is

[0040] DTW(i, j) = d ij + min{DTW(i - 1, j - 1), DTW(i - 1, j), DTW(i, j - 1)}

[0041] where the boundary condition DTW(1, 1) = d 11 , DTW(i, 0) = ∞, DTW(0, j) = ∞. Starting from the upper left corner (1, 1) of the distance matrix, calculate step by step towards the lower right corner. Finally, obtain the path from (1, 1) to (m, n), and its cumulative distance DTW(m, n) is the matching distance between the real-time action and the template action;

[0042] According to the calculated matching distance DTW(m, n), calculate the matching degree S between the real-time action and the action template. The matching degree S is obtained through normalization. The value range of the matching degree S is between [0, 1]. The closer S is to 1, the higher the matching degree.

[0043] Set a matching degree threshold T. When the matching degree S between the real-time action and a certain action template is ≥ T, the real-time action is similar to the template, and the template is automatically called, and the corresponding standard action animation is displayed in the virtual scene for the consultation experts to refer to and compare.

[0044] Optionally, in the virtual avatar driving module, for real-time action interaction with the doctor avatar, specifically, a gesture recognition device and a voice recognition device are used to dock with the virtual scene. Among them, the gesture recognition device recognizes gesture actions through a deep learning algorithm and converts them into virtual scene operation instructions, and the voice recognition device uses a deep learning-based voice recognition model to convert voice instructions into text and then execute corresponding operations.

[0045] Optionally, for the display of patient medical data in integration with the virtual scene, medical industry standard protocols of HL7 and DICOM are used for data transmission and parsing, the image data is classified and stored, and indexes are established according to the patient ID, examination time, and image type.

[0046] The present invention also proposes a real-time surgical remote video consultation method based on virtual reality, including the following steps:

[0047] Before the operation starts, the driving data acquisition module is started to collect real-time visual data, audio data, surgical instrument movement data, and patient vital sign data of the surgical scene. The collected data is fused, compressed, and encrypted by a data processing module based on deep learning, and then transmitted to the virtual avatar driving module through a high-speed network;

[0048] After receiving the data, the virtual avatar driving module reconstructs the virtual scene of the operation site, drives the actions of the virtual doctor avatar, and the consultation experts interact with the virtual scene, including perspective switching, operating the actions of the virtual doctor avatar, and querying patient information;

[0049] Monitor the actions of the surgical doctor in real time, match them with the action template library, call the corresponding template and display the standard action animation, and at the same time provide operation tips and risk warnings;

[0050] The medical image data browsing module obtains patient image data from the PACS system and the HIS system and integrates and manages it. The consultation experts collaboratively browse the images in the virtual reality environment, make annotations and share opinions;

[0051] According to the observation of the virtual scene and the consultation results, the remote consultation experts send operation instructions to the robot in the operating room through 5G communication. The robot receives the instructions and performs the corresponding surgical operation demonstration to guide the operation in real time. The force sense information during the robot operation is fed back to the consultation experts in real time. The experts adjust the operation according to the feedback and continuously monitor the robot status, and give early warnings and take measures in time when abnormalities are found.

[0052] The beneficial effects of the above technical solutions of the present invention are as follows:

[0053] 1. Multi-modal data collection and fusion enable the consultation experts to comprehensively obtain the information of the operation site, such as visual data, audio data, surgical instrument motion tracking data, and patient vital sign data, making the experts seem to be on the operation site, comprehensively perceiving the operation situation, overcoming the limitations of traditional two-dimensional video consultations, and making more accurate judgments. The virtual avatar driving module uses virtual reality technology to reconstruct a realistic operation scene. Based on the powerful functions of game development engines such as Unity or Unreal Engine, combined with high-precision models and optimized material textures, it improves the scene realism, realizes the fusion of audio and the scene, and enhances the immersion of the consultation.

[0054] 2. The motion capture technology and inverse kinematics IK algorithm are used to simulate the actions of the virtual doctor avatar. By collecting the action data of the operating doctor, preprocessing it, solving the joint target angles through the IK algorithm, and then using the animation interpolation algorithm to generate smooth transition animations, which are applied to the skeletal model, making the actions of the virtual doctor avatar highly synchronized with the operating doctor, with realistic actions and conforming to the human motion law. At the same time, physical constraints are considered to ensure natural and reasonable actions. The real-time action interaction with the doctor avatar is realized by docking the gesture recognition device and the voice recognition device with the virtual scene. The gesture recognition device uses deep learning algorithms to convert gestures into operation instructions, and the voice recognition device executes the operations after converting the voice instructions into text, facilitating experts to interact with the virtual scene during the consultation, improving the consultation efficiency and sense of participation.

[0055] 3. A large number of standard action templates are collected through the template matching module to build a library. The DTW algorithm is used to identify the real-time actions of the operating doctor and match them with the templates. According to the matching degree, it is judged whether the actions are standardized. When the matching degree is low, a deviation is prompted, and when a specific high-risk action template is matched, a warning is issued to assist the doctor in standardizing the operation and reducing the surgical risk.

[0056] 4. The data processing module calibrates the data in terms of time and space, compresses and securely encrypts the data. The calibration ensures the synchronization and consistency of multi-modal data. Image compression is based on the GAN model, audio compression uses RNN to learn the temporal features, and secure encryption uses RNN to generate keys, improving the randomness and security of the keys, preventing data leakage or tampering, protecting the security and privacy of medical data. After compression, the transmission bandwidth requirement is reduced, ensuring the efficient transmission of data.

[0057] 5. When operating in the virtual reality scene through the 5G robot assistance module, the remote consultation experts can control the robot to perform corresponding actions through 5G. The force sense information of the robot operation is real-time fed back to the experts, and the experts can adjust the operation and monitor the robot status accordingly, and give early warnings of abnormalities in a timely manner, enabling the remote experts to guide the operation in real time, combining the advantages of robot assistance, and improving the accuracy and safety of the operation. Description of the Drawings

[0058] Figure 1 This is the principle block diagram of the real-time surgical remote video consultation system based on virtual reality of the present invention;

[0059] Figure 2 This is the schematic diagram of the network connection structure of the relevant modules of the real-time surgical remote video consultation system based on virtual reality of the present invention. Specific implementation manners

[0060] To make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0061] As Figure 1 , 2 shown, a real-time surgical remote video consultation system based on virtual reality includes:

[0062] A driving data acquisition module 101, which is used to acquire visual data, audio data, surgical instrument movement tracking data, and patient vital sign data of the surgical scene.

[0063] A data processing module 102, which performs fusion processing on data from different acquisition sources, and calibrates, compresses data, and performs security encryption on visual data, audio data, surgical instrument movement data, and patient vital sign data in terms of time and space.

[0064] A virtual avatar driving module 103, which uses virtual reality technology to reconstruct the virtual scene of the surgical site, performs three-dimensional modeling and rendering on the images collected by the high-definition camera and the internal structure images collected by the medical imaging equipment, constructs a realistic surgical scene model, and based on the action tracking data of the surgeon, constructs and drives the doctor avatar in the virtual scene to perform real-time action simulation and perform real-time action interaction with the doctor avatar.

[0065] A template matching module 104, which collects and sorts out a large number of standard action templates for common surgical operations, establishes an action template library, and matches the real-time actions of the surgeon with the templates in the action template library.

[0066] A data collaboration module 105, which performs data docking with the hospital's picture archiving and communication system and HIS system, and displays patient medical data by integrating with the virtual scene.

[0067] A 5G robot assistance module 106, when performing practical operations in the virtual reality scene, remotely controls the robot through 5G to perform actions corresponding to the virtual reality scene.

[0068] Figure 2This is a schematic diagram of the network connection structure of the relevant modules of the present invention. Specifically, the present invention tracks the information of the doctor's surgical actions through limb joint trackers, head movement trackers, and surgical instrument trackers, and combines the patient's vital sign data, audio-video data, and surgical scene visual data as the data basis for constructing the virtual reality surgical system of the present invention. At the same time, data docking is achieved through the interface with the PACS system or HIS system, and through the data processing of the virtual reality server, a realistic virtual surgical scene is constructed. The specific working content and principle of each of the above modules of the present invention are as follows:

[0069] The multi-modal data collected by the driving data acquisition module 101 includes:

[0070] 1. Surgical scene visual data: A number of high-definition cameras are reasonably arranged in the operating room to capture panoramic and detailed images of the surgical area from different directions and angles, including the surgical site, instrument operation, actions of medical staff, etc., to ensure the complete acquisition of visual information during the surgical process. At the same time, high-resolution medical imaging acquisition equipment such as C-arm machines and endoscopes is equipped to collect internal structure images of the surgical site to provide more comprehensive visual materials for consultation.

[0071] 2. Audio data: A microphone array with high sensitivity and low noise is selected and installed at a suitable position in the operating room to collect the sound information at the surgical site, including the collision sound of surgical instruments, doctor's instructions, the beeping sound of patient vital sign monitoring equipment, etc., to achieve the omnidirectional capture of surgical sounds and provide more auditory clues for consultation experts.

[0072] 3. Surgical instrument motion tracking data: An inertial measurement unit (IMU) or an optical tracking marker is used to accurately track the position, posture, and motion trajectory of the surgical instrument in real time. A miniaturized and high-precision IMU sensor is installed on the surgical instrument, or an optical marker point is attached to the instrument, and through cooperation with the optical tracking system in the operating room, the motion information of the instrument in three-dimensional space is obtained, providing data support for the subsequent synchronous display and motion analysis of the instrument in the virtual scene.

[0073] 4. Patient vital sign data: By connecting with the vital sign monitoring equipment in the operating room (such as electrocardiogram monitors, blood pressure monitors, blood oxygen saturation monitors, etc.), the patient's vital sign data such as heart rate, blood pressure, blood oxygen saturation, body temperature, etc. are collected in real time and integrated in a unified data format to ensure that the patient's physiological state information can be transmitted to the consultation end in a timely and accurate manner for experts to understand the patient's situation at any time.

[0074] In this embodiment, the data processing module 102 uses deep learning algorithms to fuse data from different acquisition sources, calibrates visual data, audio data, surgical instrument motion data, and patient vital sign data in terms of time and space to ensure the synchronization and consistency of multi-modal data. Secondly, a compression model based on deep learning, an image compression algorithm based on the generative adversarial network (GAN) and an audio compression algorithm based on the recurrent neural network (RNN), are used to compress visual data and audio data, reducing the data transmission bandwidth requirements while ensuring data quality. Thirdly, a key generation model based on deep learning is adopted to dynamically generate encryption keys according to the characteristics of multi-modal data, improving the randomness and security of the keys, as follows:

[0075] (1) Calibration in terms of time and space, specifically:

[0076] Let be the i-th eigenvalue (such as pixel value, image feature, etc.) of visual data at time t, be the j-th eigenvalue (such as audio frequency, amplitude, etc.) of audio data at time t, be the k-th eigenvalue (such as instrument position coordinates, movement speed, etc.) of surgical instrument motion data at time t, be the L-th eigenvalue (such as heart rate value, blood pressure value, etc.) of patient vital sign data at time t. By constructing a multi-layer convolutional neural network, the input of the network is sequence data within a certain time window, and the output is the synchronized multi-modal data The training objective of the network is to minimize the synchronization error where is the target data after real synchronization. The convolutional layer of the CNN extracts features from the input data through the filter The formula is f is the activation function, is the input data, b c is the bias term. The pooling layer performs downsampling operations, and the fully connected layer performs data fusion and mapping to finally obtain the synchronized output.

[0077] The spatial calibration algorithm uses a deep learning object detection algorithm to detect surgical instruments and patient body parts in visual data, obtaining their coordinates in the image At the same time, the real coordinates of the instrument in three-dimensional space are obtained according to the surgical instrument motion tracking data Through the pre-established camera calibration matrix K and coordinate transformation matrix T, the coordinates in visual data are converted into spatial coordinates consistent with the motion tracking data. The formula is Then calculate the spatial calibration error Adjust the coordinate transformation matrix T by the gradient descent method to minimize the error and achieve spatial calibration.

[0078] (2) Data compression is specifically as follows:

[0079] Perform image data compression based on the GAN-based image compression model: Let G be the generator network and D be the discriminator network. For the original high-definition surgical image x, the generator network G attempts to generate a compressed image G(z), where z is a random noise vector. The task of the discriminator network D is to distinguish the real image x from the generated compressed image G(z);

[0080] The loss function of the generator network is The loss function of the discriminator network is p x (x) is the distribution of the real image, and p z (z) is the distribution of the noise vector. Through adversarial training, the generator network learns to generate a compressed image that is similar to the original image but has a smaller amount of data. In the generator network, convolutional layers and deconvolutional layers are used for image encoding and decoding operations. The encoding process formula is The decoding process formula is where is the convolutional kernel weight, and b e and b d are the bias terms, and f is the activation function.

[0081] Perform audio compression based on RNN: Let x(t) be the value of the audio signal at time t, h(t) be the hidden state of the RNN at time t, and y(t) be the compressed audio signal. The update formula of the RNN is h(t) = tanh(W hh h(t - 1) + W hx x(t) + b h ), where W hh is the weight matrix from the hidden layer to the hidden layer, W hx is the weight matrix from the input to the hidden layer, b h is the hidden layer bias term, and the output layer formula is y(t) = W hy h(t) + b y , where W hy is the weight matrix from the hidden layer to the output layer, and b y is the output layer bias term; By training the RNN to learn the temporal characteristics of the audio signal and remove redundant information, audio data compression is achieved, and the training objective is to minimize the reconstruction error is the original audio signal reconstructed from the compressed audio signal.

[0082] (3) Secure encryption is specifically as follows:

[0083] Let the sequence of feature vectors of multimodal data be X = {x1, x2, ···, x T}, where x t represents the feature vector at time t, and T is the sequence length; for time t, the update formula for the hidden layer state h t of the RNN is:

[0084] h t = tanh(W hh h t-1 + W hx x t + b h )

[0085] where W hh is the weight matrix from the hidden layer to the hidden layer, which determines the influence degree of the hidden layer state at the previous moment on the hidden layer state at the current moment; W hx is the weight matrix from the input to the hidden layer, which is used to map the input feature vector to the hidden layer space; b h is the bias term of the hidden layer; tanh is the hyperbolic tangent activation function, which is used to introduce non - linear transformation so that the RNN can learn complex feature relationships. The encryption key K is generated from the hidden layer state of the RNN, and the specific formula is:

[0086] K = W hk h T + b k

[0087] where W hk is the weight matrix from the hidden layer to the key generation layer, which is used to map the hidden layer state at the final moment to the encryption key; b k is the bias term of the key generation layer; by using the processing of the RNN on the sequence of multimodal data feature vectors, the information in the sequence is encoded into the hidden layer state, and finally the encryption key is generated;

[0088] where, during model training, with the randomness and security of the key as the objective function, the quality of the key is evaluated by calculating the entropy value H(K) of the key and the correlation Corr(K, X) with the data. The objective function J can be expressed as:

[0089] J = αH(K)+(1 - α)(1 - Corr(K, X))

[0090] Among them, α is a trade-off coefficient used to balance the importance of entropy value and correlation in the objective function. By continuously adjusting the parameters of the RNN (Wn, Wm, Wk, b, b, etc.), the generated key has higher randomness (high entropy value) and lower correlation with the original data, thereby improving the security of the key and preventing the data from being easily cracked during transmission or the key being inferred using data features. At the same time, an anomaly detection algorithm of deep learning is used to analyze the encrypted data to detect whether the data has been tampered with. Once an anomaly is found, an alarm is triggered in a timely manner to notify relevant personnel, further ensuring the security of the data.

[0091] The virtual avatar driving module 103 reconstructs the virtual scene of the surgical site using virtual reality technology, specifically based on game development engines such as Unity or Unreal Engine. Utilize its powerful graphics rendering capabilities and virtual reality support functions. During the scene modeling process, high-precision human models and surgical instrument models are adopted, and the model materials and textures are optimized according to the real surgical scene to improve the realism of the scene. The integration of audio with the virtual scene is achieved through an audio engine to ensure that the sound localization and propagation effects conform to the real scene.

[0092] Secondly, the virtual avatar driving module 103 constructs and drives the doctor avatar in the virtual scene for real-time motion simulation based on the motion tracking data of the surgeon. Specifically, motion capture technology and inverse kinematics IK algorithms are used to achieve the motion simulation of the virtual doctor avatar:

[0093] A motion capture device (such as an optical motion capture system or an inertial motion capture device) real-time obtains the motion data of the surgeon's limb joints, positions, angles, etc. Let the obtained joint data sequence be J = {j1, j2, ···, j T}, where j t represents the joint state data at time t (including the position, angle, etc. information of each joint), and T is the length of the collected time series. Preprocess the collected original joint data, including data cleaning (removing noise, outliers, etc.) and data smoothing processing to improve the data quality. Data cleaning can judge and eliminate data points that deviate significantly from the normal joint motion range by setting a reasonable threshold range. Data smoothing processing can adopt algorithms such as the moving average method or the Kalman filter. For example, when using the moving average method, for the joint angle θ t at time t, its smoothed angle is calculated through the formula where N is the moving average window size to reduce data fluctuations and make the joint motion data more continuous and stable.

[0094] Build a skeletal model of the virtual doctor image, define the connection relationships and motion constraints between the bones. For example, determine that the upper limb bones include the humerus, ulna, radius, etc., and the rotational joints and motion range limits between them.

[0095] According to the position information of the end effector (such as the hand) in the motion capture data, use the IK algorithm to solve the target angles of each joint in the skeletal model. Let the target position of the end effector in the three-dimensional space be P t , the current angle of joint i in the skeletal model is α i , the joint angle variable vector is α = {α1, α2, ···, α n} and n is the number of joints. The IK algorithm iteratively solves the optimization problem to make the calculated position P(α) of the end effector as close as possible to the target position P t , and the optimization objective function can be expressed as

[0096] E(α) = ||P(α) - P t || 2

[0097] During the solution process, use the Jacobian matrix J(α) for iterative calculation. The element J ij (α) represents the sensitivity of the speed of the end effector in the i direction to the change in the angle of joint j. According to the Newton-Raphson iteration method, the update formula for the joint angle is Δ α = J(α) + ΔP, where J(α) + is the pseudo-inverse of the Jacobian matrix, and ΔP = P t - P(α) is the position error of the end effector. Through multiple iterative calculations, the joint target angle α that meets the accuracy requirements is obtained t .

[0098] According to the obtained joint target angle α t , use an animation interpolation algorithm (such as linear interpolation or spline interpolation) to generate a smooth transition animation of the virtual doctor image from the current pose to the target pose. Let the joint angle of the current pose be α c , the transition time be T tr , for each joint i, the angle change θ i (t), t ∈ [0, T tr can be calculated through the interpolation formula, and the interpolation formula is

[0099]

[0100] Apply the generated animation data to the skeletal model of the virtual doctor image, driving the virtual doctor image to simulate the actions of the surgical doctor in real time in the virtual reality scene, achieving a high degree of synchronization and realistic simulation between the actions of the virtual doctor image and the actions of the doctor at the surgical site, enabling the consultation experts to clearly observe the details of the surgical operation, and improving the effect and quality of the consultation. At the same time, during the animation generation process, consider the physical constraints of human kinematics and dynamics, such as speed limits, acceleration limits, collision detection, etc. of joint movements, to ensure that the actions of the virtual doctor image conform to the real human movement laws and avoid unreasonable action postures. For example, by setting the maximum speed and acceleration thresholds of joint movements, constrain the calculated joint angle change rate to ensure the natural smoothness of the animation; perform collision detection in the virtual scene. When the limbs of the virtual doctor image collide with the objects in the scene, adjust the action posture in a timely manner to avoid unrealistic phenomena such as penetration.

[0101] In addition, the virtual image driving module 103 can also achieve seamless docking of the handle, gesture recognition device, and voice recognition device with the virtual scene. The button and joystick operations of the handle trigger corresponding actions in the virtual scene through the event listening mechanism. The gesture recognition device uses deep learning algorithms to recognize gesture actions and convert them into virtual scene operation instructions. The voice recognition device uses a deep learning-based voice recognition model to convert voice instructions into text and then execute corresponding operations.

[0102] The template matching module 104 collects and organizes standard action templates for a large number of common surgical operations, establishes an action template library, analyzes the real-time actions of the surgical doctor, and matches them with the templates in the action template library. It uses an action recognition algorithm based on the dynamic time warping (DTW) algorithm to match the real-time actions of the surgical doctor with the action templates. For the real-time action data of the surgical doctor, obtain information such as joint position midpoints, angles, or limb movement trajectories through action capture devices (such as inertial measurement units, optical motion capture systems, etc.), and represent them as time series data. Let the real-time action sequence be R = {r1, r2, ···, r m}

[0103] where r i represents the action feature vector at time i (such as joint angle vector or limb endpoint coordinate vector, etc.), and m is the length of the real-time action sequence. Preprocess the real-time action sequence, including data cleaning (removing outliers, noise, etc.) and normalization processing. Data cleaning can judge and correct obvious error data points by setting a reasonable threshold range. Normalization processing maps the action feature values to a specific interval. For example, normalize the joint angles to the interval [-1, 1]. The formula is

[0104]

[0105] where rij is the joint angle of the j-th at the i-th moment in the real-time action sequence, is the normalized angle value, which makes the action features of different scales and ranges comparable.

[0106] For each standard action template in the action template library, the above data representation and preprocessing operations are also performed. Let the action template sequence be T = {t1, t2, …, tn}, where tj represents the action feature vector at moment j in the template, and n is the length of the template sequence.

[0107] Calculate the distance matrix D between the real-time action sequence R and each action template sequence T. The element d of the distance matrix ij represents the distance metric between the feature vectors of the real-time action at moment i and the template action at moment j. Common distance metric methods include the Euclidean distance, and the formula is where d is the dimension of the action feature vector.

[0108] Use the DTW algorithm to search for the optimal path in the distance matrix D to minimize the cumulative distance. Let the path P = {(i1, j1), (i2, j2), ··· (i l , j l )}, where (i k , j k ) represents the k-th node in the path (the corresponding moment pair of the real-time action and the template action), and l is the length of the path. The DTW algorithm calculates the cumulative distance of the path through the dynamic programming idea, and the recurrence formula is

[0109] DTW(i, j) = d ij + min{DTW(i - 1, j - 1), DTW(i - 1, j), DTW(i, j - 1)}

[0110] where the boundary conditions are DTW(1, 1) = d 11 , DTW(i, 0) = ∞, DTW(0, j) = ∞. Starting from the upper left corner (1, 1) of the distance matrix, calculate step by step towards the lower right corner, and finally obtain the path from (1, 1) to (m, n). Its cumulative distance DTW(m, n) is the matching distance between the real-time action and the template action.

[0111] According to the calculated matching distance DTW(m, n), calculate the matching degree between the real-time action and the action template. The matching degree S can be obtained through normalization. For example where max(DTW) is the maximum possible matching distance set in advance, which can be obtained by calculating all possible action pairs. The value range of the matching degree S is between [0, 1]. The closer S is to 1, the higher the matching degree.

[0112] Set the matching degree threshold T. When the matching degree S between the real-time action and a certain action template is ≥ T, it is considered that the real-time action is similar to the template, and the template is automatically called, and the corresponding standard action animation is displayed in the virtual scene for the consultation experts to refer to and compare. At the same time, according to the matching result, the system can provide operation tips and risk warnings. For example, when the matching degree is low, it is prompted that the actions of the surgeon may be deviated and attention should be paid to the operation specifications; when a specific high-risk action template is matched, a risk warning is issued to remind the doctor to pay attention to potential risks, help the surgeon standardize the operation, and reduce the surgical risk.

[0113] The data collaboration module 105 is connected to the image storage and transmission system and HIS system of the hospital for data docking, and is integrated with the virtual scene to display the patient's medical data.

[0114] The 5G robot assistance module 106, when performing actual operations in the virtual reality scene, remotely controls the robot through 5G to execute actions corresponding to the virtual reality scene.

[0115] The data collaboration module 105 realizes data docking through the interfaces of the PACS system or HIS system, and uses medical industry standard protocols such as HL7 and DICOM for data transmission and parsing. Classify and store the image data, establish indexes according to dimensions such as patient ID, examination time, and image type for convenient quick query and retrieval. At the same time, implement the cache mechanism for image data to improve the image loading speed. At the same time, it is implemented based on WebVR or similar virtual reality collaboration frameworks, supporting multiple consultation experts to log in to the virtual environment simultaneously for image browsing. The experts can operate the image through the handle or gestures, such as zooming, rotating, translating, etc., and can use the virtual painting tool to mark on the image, and the marked information is synchronized to other consultation experts in real time to achieve collaborative diagnosis and discussion.

[0116] Based on the observation of the virtual scene and the consultation results, the remote consultation expert sends operation instructions to the 5G robot assistance module 106 in the operating room through 5G communication. The robot receives the instructions and executes the corresponding surgical operation demonstrations to guide the surgery in real time. The force sense information during the robot operation is fed back to the consultation expert in real time. The expert adjusts the operation according to the feedback and continuously monitors the robot status, and issues early warnings and takes measures in time when abnormalities are found.

[0117] The specific working method process of the above-mentioned virtual reality-based real-time surgical remote video consultation system of the present invention is as follows:

[0118] Before the operation starts, start the drive data acquisition module to collect real-time visual data, audio data, surgical instrument movement data, and patient vital sign data of the surgical scene. The collected data is fused, compressed, and encrypted by the data processing module based on deep learning, and then transmitted to the virtual image drive module through a high-speed network.

[0119] After receiving the data, the virtual avatar driving module reconstructs the virtual scene of the operation site, drives the actions of the virtual doctor avatar, and the consultation experts interact with the virtual scene, including perspective switching, operating the actions of the virtual doctor avatar, and querying patient information.

[0120] The actions of the operating doctor are monitored in real time, matched with the action template library, the corresponding template is called and the standard action animation is displayed. At the same time, operation tips and risk warnings are provided.

[0121] The medical imaging data browsing module obtains the patient's imaging data from the PACS system and the HIS system, integrates and manages it, and the consultation experts collaborate to browse the images in the virtual reality environment, make annotations and share opinions.

[0122] Based on the observation of the virtual scene and the consultation results, the remote consultation experts send operation instructions to the robot in the operating room through 5G communication. The robot receives the instructions and performs the corresponding surgical operation demonstration to guide the operation in real time. The force sense information during the robot operation is fed back to the consultation experts in real time. The experts adjust the operation according to the feedback and continuously monitor the status of the robot, and give early warnings and take measures in time when abnormalities are found.

[0123] In summary, the real-time surgical remote video consultation system based on virtual reality of the present invention has many beneficial effects, which are specifically as follows:

[0124] 1. Multi-modal data acquisition and fusion enable the consultation experts to comprehensively obtain the information of the operation site, such as visual data (operation site, instrument operation, actions of medical staff, etc.), audio data (collision sounds of surgical instruments, doctor's instructions, etc.), surgical instrument movement tracking data, and patient vital sign data, making the experts seem to be on the operation site, comprehensively perceiving the operation situation, overcoming the limitations of traditional two-dimensional video consultation, and making more accurate judgments. The virtual avatar driving module uses virtual reality technology to reconstruct a realistic operation scene. Based on the powerful functions of game development engines such as Unity or Unreal Engine, combined with high-precision models and optimized material textures, it improves the scene realism, realizes the fusion of audio and the scene, and enhances the immersion of the consultation.

[0125] 2. The action capture technology and inverse kinematics IK algorithm are adopted to realize the action simulation of the virtual doctor image. By collecting the action data of the surgical doctor, through preprocessing and solving the joint target angles by the IK algorithm, and then using the animation interpolation algorithm to generate a smooth transition animation, which is applied to the bone model, so that the actions of the virtual doctor image are highly synchronized with those of the surgical doctor, the actions are realistic and conform to the human motion law. At the same time, physical constraints are considered to ensure that the actions are natural and reasonable. The real-time action interaction with the doctor image is realized by docking the gesture recognition device and the voice recognition device with the virtual scene. The gesture recognition device uses the deep learning algorithm to convert the gesture into an operation instruction, and the voice recognition device executes the operation after converting the voice instruction into text, which is convenient for experts to interact with the virtual scene during the consultation, such as switching perspectives, operating the actions of the virtual doctor image, querying patient information, etc., improving the consultation efficiency and sense of participation.

[0126] 3. The template matching module collects a large number of standard action templates to build a library, uses the DTW algorithm to identify the real-time actions of the surgical doctor and match them with the templates, and judges whether the actions are standard according to the matching degree. When the matching degree is low, a deviation is prompted, and a warning is issued when matching a specific high-risk action template, assisting the doctor to operate standardly and reducing the surgical risk.

[0127] 4. The data processing module calibrates the data in terms of time and space, compresses and securely encrypts the data. The calibration ensures the synchronization and consistency of multi-modal data. The image compression is based on the GAN model, the audio compression uses the RNN to learn the temporal features, and the secure encryption uses the RNN to generate the key, improving the randomness and security of the key, preventing data leakage or tampering, protecting the security and privacy of medical data. After compression, the transmission bandwidth requirement is reduced, ensuring the efficient transmission of data.

[0128] 5. When operating in the virtual reality scene with the assistance of the 5G robot, the remote consultation expert can control the robot to perform corresponding actions through 5G. The force sense information of the robot operation is fed back to the expert in real time. The expert adjusts the operation accordingly and monitors the state of the robot, and warns of abnormalities in a timely manner, enabling the remote expert to guide the operation in real time, combining the advantages of robot assistance, and improving the accuracy and safety of the operation.

[0129] The above is the preferred implementation mode of the present invention. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle described in the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A real-time surgical remote video consultation system based on virtual reality, characterized in that: include: Drive data acquisition module, used to collect surgical scene visual data, audio data, surgical instrument movement tracking data, and patient vital signs data; The data processing module integrates data from different acquisition sources, calibrates, compresses, and securely encrypts visual data, audio data, surgical instrument motion data, and patient vital signs data in time and space; The virtual image driving module uses virtual reality technology to reconstruct the virtual scene of the surgical site, and performs three-dimensional modeling and rendering of the images collected by high-definition cameras and internal structure images collected by medical imaging equipment to build a realistic surgical scene model. Based on the motion tracking data of the surgeon, it builds and drives the doctor image in the virtual scene to perform real-time motion simulation and interact with the doctor image in real-time motion. Template matching module collects and organizes standard action templates for common surgical operations, establishes an action template library, and matches the surgeon's real-time actions with the templates in the action template library; The data collaboration module connects data with the hospital's image storage and transmission system and HIS system, and integrates with the virtual scene to display patient medical data; The 5G robot assistance module remotely controls the robot through 5G to perform actions corresponding to the virtual reality scene when performing actual operations in the virtual reality scene.

2. The real-time surgical remote video consultation system based on virtual reality according to claim 1 is characterized in that: The data processing module performs fusion processing on the data from different acquisition sources, and calibrates the visual data, audio data, surgical instrument motion data and patient vital sign data in time and space, specifically: set up is the i-th eigenvalue of the visual data at time t, is the j-th eigenvalue of the audio data at time t, is the kth eigenvalue of the surgical instrument motion data at time t, is the Lth eigenvalue of the patient's vital signs data at time t. By constructing a multi-layer convolutional neural network, the input of the network is Sequence data within a certain time window is output as synchronized multimodal data The training goal of the network is to minimize the synchronization error in To obtain the real synchronized target data, the convolutional layer of CNN passes through the filter Feature extraction is performed on the input data, and the formula is f is the activation function, is the input data, b c is the bias term, the pooling layer performs downsampling, and the fully connected layer performs data fusion and mapping, and finally obtains the synchronized output; Use deep learning object detection algorithms to detect surgical instruments and patient body parts in visual data and obtain their coordinates in the image At the same time, the real coordinates of the instrument in three-dimensional space are obtained based on the motion tracking data of the surgical instrument. Through the pre-established camera calibration matrix K and coordinate transformation matrix T, the coordinates in the visual data are converted into spatial coordinates consistent with the motion tracking data. The formula is: Then calculate the spatial calibration error The coordinate transformation matrix T is adjusted by the gradient descent method to minimize the error and achieve spatial calibration.

3. The real-time surgical remote video consultation system based on virtual reality according to claim 1 is characterized in that: The data processing module performs fusion processing on the data from different acquisition sources and compresses the visual data, audio data, surgical instrument motion data and patient vital sign data in time and space. Specifically: Image data compression based on GAN image compression model: Let G be the generator network and D be the discriminator network. For the original high-definition surgical image x, the generator network G attempts to generate a compressed image G(z), z is a random noise vector, and the task of the discriminator network D is to distinguish between the real image x and the generated compressed image G(z); The loss function of the generator network is The loss function of the discriminator network is p x (x) is the distribution of real images, p z (z) is the distribution of the noise vector. Through adversarial training, the generator network learns to generate compressed images that are similar to the original images but with smaller data volumes. In the generator network, convolutional layers and deconvolutional layers are used for encoding and decoding operations of the image. The encoding process formula is: The decoding process formula is: in is the convolution kernel weight, b e 、b d is the bias term, f is the activation function; Audio compression based on RNN: Let x(t) be the value of the audio signal at time t, h(t) be the hidden state of RNN at time t, y(t) be the compressed audio signal, and the update formula of RNN is h(t) = tanh(W hh h(t-1)+W hx x(t)+b h ), where W hh is the weight matrix from hidden layer to hidden layer, W hx is the weight matrix input to the hidden layer, b h is the hidden layer bias term, and the output layer formula is y(t) = W hy h(t)+b y , where W hy is the weight matrix from the hidden layer to the output layer, b y is the output layer bias term; by training RNN to learn the temporal characteristics of audio signals, remove redundant information, and achieve audio data compression. The training goal is to minimize the reconstruction error The original audio signal is reconstructed from the compressed audio signal.

4. The real-time surgical remote video consultation system based on virtual reality according to claim 1, characterized in that: The data processing module performs fusion processing on the data from different acquisition sources, and securely encrypts the visual data, audio data, surgical instrument motion data and patient vital signs data in time and space, specifically: Assume that the feature vector sequence of multimodal data is X = {x1, x2, ···, x T }, where x t represents the feature vector at time t, T is the sequence length; for time t, the hidden layer state h of RNN t The update formula is: h t =tanh(W hh h t-1 +W hx x t +b h ) Among them, W hh is the weight matrix from hidden layer to hidden layer, which determines the influence of the hidden layer state at the previous moment on the hidden layer state at the current moment; W hx is the weight matrix input to the hidden layer, which is used to map the input feature vector to the hidden layer space; b h is the bias term of the hidden layer; tanh is the hyperbolic tangent activation function, which is used to introduce nonlinear transformations so that RNN can learn complex feature relationships; The encryption key K is generated from the hidden layer state of the RNN, and the specific formula is: K=W hk h T +b k Among them, W hk is the weight matrix from the hidden layer to the key generation layer, which is used to map the hidden layer state at the final moment to the encryption key: k It is the bias term of the key generation layer. RNN is used to process the multimodal data feature vector sequence, encode the information in the sequence into the hidden layer state, and finally generate the encryption key. Among them, when training the model, the randomness and security of the key are taken as the objective function, and the key quality is evaluated by calculating the entropy value H(K) of the key and the correlation Corr(K,X) with the data. The objective function J is expressed as: J=αH(K)+(1-α)(1-Corr(K,X)) Among them, α is a trade-off coefficient, which is used to balance the importance of entropy and correlation in the objective function.

5. The real-time surgical remote video consultation system based on virtual reality according to claim 1 is characterized in that: The virtual image driving module uses virtual reality technology to reconstruct the virtual scene of the surgical site, and is specifically built based on the Unity or Unreal Engine game development engine.

6. The real-time surgical remote video consultation system based on virtual reality according to claim 1, characterized in that: The virtual image driving module constructs and drives the doctor image in the virtual scene to perform real-time motion simulation based on the surgeon's motion tracking data. Specifically, motion capture technology and inverse dynamics IK algorithm are used to achieve motion simulation of the virtual doctor image: Assume that the target position of the end effector in three-dimensional space is P t , the current angle of joint i in the skeleton model is α i , the joint angle variable vector is α={α1,α2,···,α n }, n is the number of joints. The IK algorithm solves the optimization problem iteratively to make the calculated position P(α) of the end effector close to the target position P t , the optimization objective function is E(α)=||P(α)-P t || 2 ; During the solution process, the Jacobian matrix J(α) is used for iterative calculation. The element J of the Jacobian matrix ij (α) represents the sensitivity of the end effector velocity in direction i to the change of joint angle j. According to the Newton-Raphson iteration method, the update formula of the joint angle is Δ α = J(α) + ΔP, where J(α) + is the pseudo-inverse of the Jacobian matrix, ΔP = P t -P(α) is the position error of the end effector. Through multiple iterative calculations, the joint target angle α that meets the accuracy requirements is obtained. t ; According to the joint target angle α obtained by solving t , the animation interpolation algorithm is used to generate a smooth transition animation of the virtual doctor image from the current posture to the target posture, assuming that the joint angle of the current posture is α c , the transition time is T tr , for each joint i, the angle change θ during the transition time i (t), t∈[0,T tr ]Calculated by interpolation formula: The generated animation data is applied to the skeleton model of the virtual doctor image, driving the virtual doctor image to simulate the surgeon's movements in real time in the virtual reality scene.

7. The real-time surgical remote video consultation system based on virtual reality according to claim 1 is characterized in that: The matching with the template in the action template library is performed, and the action recognition algorithm of the dynamic time warping DTW algorithm is used to match the real-time action of the surgeon with the action template, specifically: Calculate the distance matrix D between the real-time action sequence R and each action template sequence T, and the element d of the distance matrix ij It represents the distance measure between the feature vectors of the real-time action at time i and the template action at time j. The formula is: Where d is the dimension of the action feature vector; Assume that path P = {(i1, j1), (i2, j2), ··· (i l ,j l )}, where (i k ,j k ) represents the kth node in the path, l is the path length, and the DTW algorithm calculates the cumulative distance of the path through dynamic programming. The recursive formula is: DTW(i,j)=d ij +min{DTW(i-1,j-1),DTW(i-1,j),DTW(i,j-1)} Among them, the boundary condition DTW(1,1)=d 11 , DTW(i,0)=∞, DTW(0,j)=∞, start calculating from the upper left corner (1,1) of the distance matrix, and gradually move toward the lower right corner, and finally get the path from (1,1) to (m,n). Its cumulative distance DTW(m,n) is the matching distance between the real-time action and the model action; According to the calculated matching distance DTW(m,n), the matching degree S between the real-time action and the action template is calculated. The matching degree S is obtained through normalization processing. The value range of the matching degree S is between [0,1]. The closer S is to 1, the higher the matching degree. A matching threshold T is set. When the matching degree S ≥ T between the real-time action and a certain action template, the real-time action is similar to the template. The template is automatically called and the corresponding standard action animation is displayed in the virtual scene for reference and comparison by the consulting experts.

8. The real-time surgical remote video consultation system based on virtual reality according to claim 1, characterized in that: In the virtual image driving module, real-time action interaction is performed with the doctor image, and specifically, gesture recognition equipment and voice recognition equipment are used to connect with the virtual scene. The gesture recognition device recognizes gestures through a deep learning algorithm and converts them into virtual scene operation instructions. The voice recognition device uses a deep learning-based voice recognition model to convert voice instructions into text and then execute the corresponding operations.

9. The real-time surgical remote video consultation system based on virtual reality according to claim 1, characterized in that: The patient's medical information is displayed by integrating with the virtual scene, and the medical industry standard protocols of HL7 and DICOM are used for data transmission and analysis. The image data are classified and stored, and indexes are established according to patient ID, examination time, and image type.

10. A real-time remote surgical video consultation method based on virtual reality, characterized in that: The following steps are involved: Before the operation begins, the driving data acquisition module is started to collect the visual data of the surgical scene, audio data, surgical instrument movement data and patient vital signs data in real time. The collected data is fused, compressed and encrypted by the data processing module based on deep learning, and then transmitted to the virtual image driving module through a high-speed network; After receiving the data, the virtual image driving module reconstructs the virtual scene of the surgical site and drives the virtual doctor image to move. The consulting expert interacts with the virtual scene, including switching perspectives, operating the virtual doctor image and querying patient information. Monitor surgeons' movements in real time, match them with the action template library, call corresponding templates and display standard action animations, while providing operation prompts and risk warnings; The medical image data browsing module obtains and integrates patient image data from the PACS system and HIS system. Consulting experts can collaboratively browse images, annotate and share opinions in a virtual reality environment. Based on the virtual scene observations and consultation results, the remote consultation expert sends operation instructions to the robot in the operating room through 5G communication. The robot receives the instructions and performs the corresponding surgical operation demonstration to guide the operation in real time. The force information during the robot operation is fed back to the consulting expert in real time. The expert adjusts the operation based on the feedback and continuously monitors the status of the robot. If any abnormality is found, timely warnings will be issued and measures will be taken.

Images