A virtual reality-based telemedicine interaction method

Abstract

The application discloses a remote medical interactive method based on virtual reality, and particularly relates to the field of remote medical interaction, and comprises the following steps: through synchronous acquisition of patient end multi-modal tactile data and transmission state parameters, a joint feature vector is established, and transmission delay of each modal tactile signal is predicted in real time. Through real-time image extraction of the doctor end, the operation stage and the tactile signal sensitivity index are extracted, the reward value is calculated by combining the double-branch regression network, the mode transmission strategy is updated, and a dynamic priority allocation matrix is generated; based on the matrix, the real-time network state and the delay prediction are combined, the bandwidth slice strategy is optimized, and signal compensation is performed; in addition, the operation instrument switching and pathological mutation are detected, the priority allocation is dynamically adjusted, and efficient transmission and operation feedback of the tactile signal are ensured.

Description

Technical Field

[0001] The present invention relates to the field of telemedicine interaction technology, and more particularly, to a telemedicine interaction method based on virtual reality. Background Art

[0002] In virtual reality telemedicine scenarios, when doctors perceive patient vital signs in real time through tactile feedback devices, they need to rely on the synchronous transmission and rendering mechanism of multimodal tactile signals such as temperature, vibration, and pressure to restore the real sense of touch. However, existing technologies are limited by the differences in the physical characteristics of tactile signals and defects in network resource allocation mechanisms. Existing systems use a fixed priority transmission strategy. For example, the pressure signal is preset as the highest priority and exclusively occupies the low-latency channel. However, in scenarios such as burn wound palpation and inflammation area inspection, the temperature signal is more valuable for lesion identification than the pressure signal. The fixed strategy forcibly suppresses the transmission of high-value modal signals, resulting in the loss of key diagnostic information. When network bandwidth fluctuates, there is a lack of cross-modal coordination mechanism, and the signal priority cannot be dynamically adjusted according to the stage of surgery. The high-latency signal queue is blocked, resulting in a break in the spatiotemporal continuity of tactile perception. Doctors are unable to construct a coherent pathological feature map through tactile feedback, which directly affects the reliability and diagnostic efficiency of telemedicine.

[0003] Therefore, how to dynamically adjust the transmission strategy priority of multimodal tactile signals and then adjust the transmission bandwidth of modal transmission signals to achieve a telemedicine interaction method for virtual reality scenarios based on modal signal control is an urgent problem that needs to be solved.

[0004] In order to solve the above problems, a technical solution is now provided. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a remote medical interaction method based on virtual reality to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] S1: Synchronously collect multimodal tactile data and corresponding transmission state parameters of the patient-side device, and establish a joint feature vector of the fused modal tactile data and signal transmission state;

[0008] S2: Establish a multimodal transmission delay prediction model based on the joint feature vector and output the transmission delay prediction value of each modality tactile signal in real time;

[0009] S3: identifying the real-time image of the doctor's virtual reality device, extracting the surgical stage corresponding to the image and the diagnostic sensitivity index of each modality tactile signal, and simultaneously establishing a double-branch regression network to obtain a multi-dimensional reward value of the effectiveness of the tactile information, and the reward value is back-propagated to the modality transmission strategy to generate a dynamic priority allocation matrix of the modality tactile signal;

[0010] S4: based on the priority of the dynamic priority allocation matrix of the modality tactile signal, dividing the virtual channel of the modality, and according to the real-time network state of the virtual channel, combining the real-time transmission delay prediction value, taking the minimum transmission delay of the modality tactile signal as the target, updating the bandwidth slice strategy through Q value iteration;

[0011] S5: establishing a cross-modality unified time coordinate system, calculating the time offset of each modality tactile signal relative to the reference axis and compensating the signal;

[0012] S6: detecting the switching event of the surgical instrument type and the mutation state of the pathological characteristics, and generating the dynamic priority allocation matrix of the modality tactile signal after the scene changes.

[0013] In a preferred embodiment, in S1, the multi-modal tactile data of the patient end device and the corresponding transmission state parameters are synchronously collected, and the joint feature vector fusing the modality tactile data and the signal transmission state is established, which specifically includes:

[0014] The multi-modal tactile data of each tactile sensor of the patient end device and the transmission state parameters of the corresponding modality tactile signal are synchronously collected by using multi-threading, and a multi-modal tactile signal original data set carrying a unified timestamp is generated, wherein the multi-modal tactile signal includes pressure, temperature and vibration tactile signals;

[0015] The time domain features of the multi-modal tactile signal are extracted, and the signal transmission state parameter features are extracted by using a long short-term memory network, and a joint feature vector fusing the modality tactile data and the signal transmission state is established.

[0016] In a preferred embodiment, in S2, a multi-modal transmission delay prediction model is established based on the joint feature vector, and the transmission delay prediction value of each modality tactile signal is output in real time, which specifically includes:

[0017] The joint feature vector is input into a fully connected neural network, and a modality transmission delay prediction model of pressure, temperature and vibration modalities is established through end-to-end supervised learning, and the loss function adopts a weighted combination of root mean square error and prediction value variance;

[0018] The real-time delay observation value of each modality tactile signal transmission is input to dynamically adjust the modality transmission delay prediction model until the model prediction accuracy reaches the set index;

[0019] Based on the adjusted model, the transmission delay prediction value of each modal tactile signal is output in real time.

[0020] In a preferred embodiment, in S3, the real-time image of the virtual reality device on the doctor's side is identified, the surgical stage of the disease corresponding to the image and the diagnostic sensitivity index of each modal tactile signal are extracted, and a two-branch regression network is established to obtain a multi-dimensional reward value for the effectiveness of the tactile information. The reward value is back-propagated to the modal transmission strategy to generate a dynamic priority allocation matrix for the modal tactile signal. Specifically, the following steps are included:

[0021] Identify the real-time image of the doctor's virtual reality device, obtain the type of surgical instrument and the pathological status of the virtual operating area, and simultaneously search the historical pathology database to extract the surgical stage of the corresponding disease and the tactile signal diagnostic sensitivity index;

[0022] The surgical stage is encoded as a discrete state variable, and the pathological state is mapped to a high-dimensional feature vector. These are input into the tensor fusion layer together with the signal sensitivity index to generate a multi-dimensional joint state space representation of the reinforcement learning model.

[0023] A dual-branch regression network was established. The main branch received real-time tactile feedback scores from the doctor, while the auxiliary branch analyzed the frequency of pauses during surgery. The two branches jointly output a multi-dimensional reward value for the effectiveness of the tactile information.

[0024] Based on the deep deterministic policy gradient algorithm, with the joint state space as input and QoS level assignment as action space, the reward value is back-propagated to the modal transmission strategy to generate a dynamic priority assignment matrix of the modal tactile signals.

[0025] In a preferred embodiment, in S4, based on the priority sorting of the dynamic priority allocation matrix of the modal tactile signal, the modal virtual channels are divided, and according to the real-time network status of the virtual channels and the real-time transmission delay prediction value, with the goal of minimizing the transmission delay of the modal tactile signal, the bandwidth slicing strategy is iteratively updated through the Q value, specifically including:

[0026] Based on the dynamic priority allocation matrix of modal tactile signals, the SDN controller divides independent virtual channels, establishes a mapping relationship between tactile modalities and virtual channels, and allocates initial bandwidth share according to priority order;

[0027] Collect the real-time throughput, packet loss rate, and transmission delay prediction values ​​of each virtual channel to build a network status matrix containing spatiotemporal characteristics;

[0028] Taking the network state matrix as input and the bandwidth allocation action as output, the bandwidth slicing strategy is updated through Q value iteration with the goal of minimizing the transmission delay of the modal tactile signal.

[0029] Monitor whether the transmission delay of the modal tactile signal with the highest priority increases after the bandwidth slice is updated. If so, trigger a policy rollback.

[0030] In a preferred embodiment, in S5, establishing a cross-modal unified time coordinate system, calculating the time offset of each modal tactile signal relative to the reference axis, and performing signal compensation specifically include:

[0031] The pressure signal timestamp is selected as the reference axis to establish a cross-modal unified time coordinate system;

[0032] Based on the reference axis time coordinate system, the timestamp sequences of the temperature and vibration signals are analyzed respectively, and the time offset of each modal tactile signal relative to the reference axis is calculated by linear regression method.

[0033] In view of the temperature signal delay, a cubic spline interpolation algorithm is used in the receiving end buffer to generate a continuous signal sequence that is spatially and temporally aligned with the pressure signal using the reference axis time as the independent variable.

[0034] According to the time offset of the vibration signal, a finite impulse response digital filter is established to perform phase correction on the vibration signal.

[0035] The time domain correlation coefficient and spatial coordinate deviation of the compensated temperature, vibration signal and pressure signal are calculated, and the compensation is terminated when both indicators reach the preset tolerance threshold.

[0036] In a preferred embodiment, in S6, detecting the surgical instrument type switching event and the pathological feature mutation state, and generating a dynamic priority allocation matrix of the modal tactile signal after the scene change specifically includes:

[0037] Identify the real-time image of the doctor's virtual reality device, detect surgical instrument type switching events and pathological feature mutation states, and generate scene change trigger signals;

[0038] When a scene change trigger signal appears, the historical pathology database is queried to extract the surgical stage of the corresponding disease and the tactile signal diagnostic sensitivity index, and the multidimensional joint state space is reconstructed;

[0039] A sliding time window mechanism is used to dynamically collect time series data of tactile signals of each modality after scene changes;

[0040] Based on the temporal modal tactile signal data after the scene change, the multi-dimensional reward value of the tactile information effectiveness is recalculated, and a dynamic priority allocation matrix of the modal tactile signals after the scene change is generated.

[0041] On the other hand, the present invention provides a virtual reality-based telemedicine interaction system, comprising a delay prediction module, a tactile signal priority allocation module, a bandwidth strategy update module, a signal compensation module, and an event switching module:

[0042] Delay prediction module: by fusing the joint features of multi-modal tactile data and signal transmission state, a modal transmission delay prediction model of pressure, temperature and vibration modal is established, and the transmission delay prediction value of each modal tactile signal is output in real time;

[0043] Tactile signal priority allocation module: the real-time image of the doctor's virtual reality device is recognized, the type of surgical instrument and the pathological state of the virtual operation area are obtained, and a double-branch regression network is established to output a multi-dimensional reward value of the effectiveness of tactile information, and the reward value is back propagated to the modal transmission strategy to generate a dynamic priority allocation matrix of the modal tactile signal;

[0044] Bandwidth strategy update module: based on the priority matrix of the modal tactile signal, the SDN controller divides the virtual channel and allocates the initial bandwidth. A network state matrix is constructed, the matrix is taken as the input, the bandwidth slicing strategy is optimized to minimize the transmission delay, and the delay change of the signal with the highest priority is monitored, and if it rises, the strategy is rolled back;

[0045] Signal compensation module: a unified time coordinate system is established with the pressure signal as the reference axis, the time offset of the temperature and vibration signals is calculated, and the three spline interpolation and the finite impulse response filter are used for compensation respectively, and finally the alignment effect is checked according to the time domain correlation and spatial deviation;

[0046] Event switching module: detect the surgical instrument type switching event and the pathological feature mutation state, if the scene change trigger signal appears, generate the dynamic priority allocation matrix of the modal tactile signal after the scene change through the tactile signal priority allocation module.

[0047] The technical effects and advantages of the remote medical interactive method based on virtual reality of the application are as follows:

[0048] Through the joint modeling of multi-modal tactile data and network transmission state, the cross-domain perception ability is constructed, and the rigid defects of the traditional fixed priority strategy are broken. The dynamic priority allocation mechanism based on the double-branch regression network and reinforcement learning realizes the adaptive coupling of the value weight of the tactile signal and the network health degree, and combines the virtual channel mapping and the bandwidth slicing strategy driven by Q learning, which dynamically balances the multi-modal resource competition while ensuring the low delay transmission of high sensitivity signals. The cross-modal time offset compensation algorithm eliminates the sensory tearing phenomenon by unifying the space-time reference axis, and improves the space-time consistency of tactile feedback. The incremental priority update mechanism triggered by scene events realizes the precise adaptation of the surgical stage and the evolution process of pathological features, dynamically adjusts the signal transmission strategy through instrument switching detection and pathological mutation recognition, and ensures the lossless transmission of key diagnostic information in complex scenes. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1This is a schematic diagram of a virtual reality-based telemedicine interaction method of the present invention;

[0050] Figure 2 This is a structural diagram of a virtual reality-based telemedicine interactive system of the present invention. DETAILED DESCRIPTION

[0051] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] Example 1

[0053] Figure 1 The present invention provides a remote medical interaction method based on virtual reality, which includes the following steps:

[0054] S1: Synchronously collect multimodal tactile data and corresponding transmission state parameters of the patient-side device, and establish a joint feature vector of the fused modal tactile data and signal transmission state;

[0055] S2: Establish a multimodal transmission delay prediction model based on the joint feature vector and output the transmission delay prediction value of each modality tactile signal in real time;

[0056] S3: Identify the real-time image of the doctor's virtual reality device, extract the surgical stage of the disease corresponding to the image and the diagnostic sensitivity index of each modal tactile signal, and simultaneously establish a two-branch regression network to obtain a multi-dimensional reward value for the effectiveness of the tactile information. The reward value is back-propagated to the modal transmission strategy to generate a dynamic priority allocation matrix for the modal tactile signals.

[0057] S4: Based on the priority sorting of the dynamic priority allocation matrix of the modal tactile signals, the modal virtual channels are divided. Based on the real-time network status of the virtual channels and the real-time transmission delay prediction value, the bandwidth slicing strategy is updated through Q value iteration with the goal of minimizing the transmission delay of the modal tactile signals.

[0058] S5: Establish a cross-modal unified time coordinate system, calculate the time offset of each modal tactile signal relative to the reference axis and perform signal compensation;

[0059] S6: Detect surgical instrument type switching events and pathological feature mutation states, and generate a dynamic priority allocation matrix for modal tactile signals after scene changes.

[0060] In S1, the multimodal tactile data and corresponding transmission state parameters of the patient-side device are synchronously collected to establish a joint feature vector of the fused modal tactile data and the signal transmission state.

[0061] On the patient-side device, a multimodal tactile sensor array is used to synchronously collect tactile signal data of different modalities through multi-threading. The multimodal tactile signals include pressure, temperature and vibration tactile signals. At the same time, the signal transmission state parameters are recorded in real time. The transmission state parameters include transmission delay, data loss rate, available channel bandwidth and transmission jitter information. All collected data are attached with high-precision timestamps to ensure the synchronization of different modal data within the same time window.

[0062] During the acquisition process, to address the asynchrony issues of different sensors, a high-precision clock synchronization mechanism is adopted. The main thread coordinates the multi-threaded data acquisition process of each sensor, so that all modal tactile data and transmission status parameters are synchronously stored at a sampling rate of no less than 1000Hz. Data buffering is performed based on a message queue management mechanism to prevent data loss caused by short-term network jitter or data transmission anomalies, thereby constructing a multimodal tactile signal raw data set with a unified timestamp.

[0063] After obtaining a complete dataset of raw tactile signal data, we first perform time domain feature extraction on tactile signals of different modalities. For example, the pressure signal extracts contact duration, pressure change rate, and peak pressure value; the temperature signal extracts steady-state temperature, temperature change gradient, and maximum temperature change rate; and the vibration signal extracts vibration frequency, amplitude, and energy distribution. All feature data are then normalized.

[0064] At the same time, a long short-term memory network is used to extract features of the transmission state parameters of tactile signals and construct a timing feature model to predict the stability of future signal transmission. The input data includes transmission delay, data loss rate, bandwidth and jitter parameters at current and historical times. The model output is the signal transmission state trend in the future period of time, and the confidence of each modal tactile signal under different transmission states is calculated, thereby establishing a joint feature vector that integrates multimodal tactile data and signal transmission state information.

[0065] In S2, a multimodal transmission delay prediction model is established based on the joint feature vector, and the transmission delay prediction value of each modal tactile signal is output in real time.

[0066] A multi-layer, fully connected neural network is constructed, consisting of an input layer, multiple hidden layers, and an output layer. The number of neurons in each layer is optimized based on the feature dimension and model complexity. The network uses ReLU as the activation function, and the output layer uses linear activation to predict modal transmission delay using a regression approach.

[0067] During training, the loss function is a weighted sum of the root mean square error (RMS) and the variance of the predicted values, aiming to simultaneously minimize prediction error and improve model stability. The Adam optimizer is used for gradient updates, with an initial learning rate of 0.001 and dynamic adjustment every 1000 epochs to ensure rapid model convergence. The training data is split into an 80% training set and a 20% test set, and hyperparameters are optimized through cross-validation.

[0068] Adopt online learning strategy to incrementally adjust the weights of neural network. The specific method is as follows:

[0069] (1) Feed the error back to the loss function, calculate the gradient and perform a mini-batch update;

[0070] (2) Update the training data buffer and retrain some network layers in a timely manner to improve prediction accuracy. During the online adjustment process, dynamically adjust the root mean square error and variance weight ratio to ensure that the model still has stable prediction capabilities in a time-varying environment. Stop model adjustment and enter the stable prediction stage until the prediction accuracy reaches 95% or above, or the error converges stably to the set range (e.g., within 0.01 seconds).

[0071] Based on the adjusted model, the transmission delay prediction value of each modal tactile signal is output in real time.

[0072] In S3, the real-time image of the virtual reality device on the doctor's side is identified, and the surgical stage of the disease corresponding to the image and the diagnostic sensitivity index of each modal tactile signal are extracted. At the same time, a two-branch regression network is established to obtain the multi-dimensional reward value of the effectiveness of tactile information. The reward value is back-propagated to the modal transmission strategy to generate a dynamic priority allocation matrix for modal tactile signals.

[0073] By analyzing the real-time images of the doctor's VR device and using a deep convolutional neural network (CNN), the surgical images are segmented and classified to extract relevant information about surgical instruments and pathological areas. For example, the currently used surgical instruments (such as electrosurgery and ultrasonic scalpels) and the pathological characteristics of the patient's virtual tissue (such as tumor hardness and burn area temperature) are identified. The historical case database is also searched simultaneously to obtain typical surgical stage divisions for similar diseases (such as "tumor resection stage") and diagnostic sensitivity indicators of various tactile signals (pressure, temperature, etc.). The historical case database mainly refers to historical surgical imaging data.

[0074] The diagnostic sensitivity index is specifically set based on the accuracy requirements of the tactile signals of each modality during the surgical stage. For example, when the system recognizes that the doctor is using an "ultrasonic knife" to operate in the liver tumor area, it automatically retrieves historical records from the database, determines that it is currently in the "tumor resection stage", and loads the sensitivity of the pressure signal in this stage as 0.88 (high priority) and the temperature signal as 0.35 (low priority).

[0075] The discrete surgical stages (such as "hemostasis stage") are fused with continuous pathological features (such as tissue elasticity value and temperature distribution), combined with the tactile signal sensitivity index, to construct the state input of the reinforcement learning model. For example, when the surgical stage is encoded as "hemostasis stage", the corresponding encoding is [0, 0, 0, 1, 0, 0].

[0076] In addition, the diagnostic sensitivity index of the tactile signal is also integrated as a feature. Based on this, a tensor fusion layer is used to combine this feature information with the tactile signal sensitivity index to generate a multidimensional joint state space representation that comprehensively describes the real-time state information during surgery. For example, the pathological area has a temperature gradient of 0.8 and a pressure signal-to-noise ratio of 32dB. The sensitivity indexes are 0.75 for pressure and 0.20 for temperature. The concatenated state vector is [0, 0, 0, 1, 0, 0, 0.8, 32, 0.75, 0.20]. This feature vector is then input into the reinforcement learning model.

[0077] A two-branch regression network is established. The main branch receives the doctor's real-time tactile feedback score, and the auxiliary branch analyzes the frequency of operation pauses during surgery. The two branches jointly output a multi-dimensional reward value for the effectiveness of the tactile information. The following is a specific example:

[0078] Main branch: If the doctor scores the tactile feedback 4.5 / 5 (high quality) and the signal-to-noise ratio is 40dB (high definition), the corresponding output reward is +0.8.

[0079] Auxiliary branch: If the instrument pauses twice per minute during surgery (smooth operation), the corresponding output reward is +0.3.

[0080] Based on the deep deterministic policy gradient algorithm, with the joint state space as input and the QoS level assignment as the action space, the reward value is back-propagated to the modal transmission strategy to generate a dynamic priority assignment matrix for the modal tactile signals. The matrix assigns a priority weight to each tactile signal according to the different stages of the surgery and the current pathological state.

[0081] In S4, the modal virtual channels are divided based on the priority sorting of the dynamic priority allocation matrix of the modal tactile signals. The bandwidth slicing strategy is updated through Q value iteration according to the real-time network status of the virtual channels and the real-time transmission delay prediction value, with the goal of minimizing the transmission delay of the modal tactile signals.

[0082] Based on the dynamic priority allocation matrix of modal haptic signals, a software-defined networking (SDN) controller is used to divide the system into multiple independent virtual channels. Bandwidth is allocated to each virtual channel based on the priority of the modal haptic signals. Initial bandwidth allocation is performed based on the priority order of the modal haptic signals. Specifically, the initial transmission bandwidth (originally set by the device) is weighted and adjusted based on the priority weights to form the new initial bandwidth allocation.

[0083] The real-time throughput, packet loss rate, and transmission delay prediction values ​​of each virtual channel are collected to construct a network status matrix containing spatiotemporal characteristics, forming a dynamic network status representation and providing a reference input basis for bandwidth allocation strategy.

[0084] The network state matrix is ​​used as input, bandwidth allocation actions are output, and the optimization goal is to minimize the transmission delay of modal tactile signals. A Q-learning algorithm based on reinforcement learning adjusts the bandwidth slicing strategy through iterative updates of Q values.

[0085] After each bandwidth allocation strategy update, bandwidth resource allocation is adjusted based on Q-value feedback to minimize delays and improve transmission efficiency when tactile signals are transmitted in the network. The specific comparison method for minimizing the transmission delay of modal tactile signals is to compare the maximum reduction ratio of the actual transmission delay observed for each modal channel compared to the input transmission delay prediction value, rather than comparing the actual transmission delay of each modality.

[0086] At the same time, in the process of minimizing the transmission delay of the modal tactile signal, if there is a bandwidth allocation conflict between the modalities, the transmission bandwidth of the high-priority modal signal will be allocated first according to the priority order.

[0087] Monitor whether the transmission delay of the modal tactile signal with the highest priority increases after the bandwidth slice is updated. If so, trigger a policy rollback. When the transmission delay of the modal tactile signal with the highest priority increases, the surface bandwidth allocation policy completely fails. At this time, the policy needs to be rolled back and readjusted.

[0088] In S5 , a cross-modal unified time coordinate system is established, the time offset of each modal tactile signal relative to the reference axis is calculated, and signal compensation is performed.

[0089] The pressure signal's timestamp is selected as the reference axis to establish a unified time coordinate system across all modalities. The pressure signal's timestamp is set as the origin of the global time coordinate system, and the timestamps of other modalities (such as temperature and vibration signals) are aligned based on this reference axis.

[0090] After the cross-modal time coordinate system is established, the timestamp sequences of the temperature signal and vibration signal are analyzed separately. Through the linear regression method, the system calculates the time offset of the temperature signal and vibration signal relative to the reference axis of the pressure signal.

[0091] In view of the temperature signal delay, the cubic spline interpolation algorithm is used in the receiving end buffer to generate a compensated temperature signal by constructing a smooth curve between known data points. The reference axis time is used as the independent variable to generate a continuous signal sequence that is temporally and spatially aligned with the pressure signal.

[0092] According to the time offset of the vibration signal, a finite impulse response digital filter is established to perform phase correction on the vibration signal. The FIR filter can effectively process the phase difference of the signal and adjust the phase of the vibration signal to fully align it with the reference time of the pressure signal.

[0093] The time-domain correlation coefficient and spatial coordinate deviation of the compensated temperature, vibration, and pressure signals are calculated. The time-domain correlation coefficient is used to measure the synchronization of the signals, while the spatial coordinate deviation is used to evaluate the degree of signal offset in space. Compensation ends when both indicators reach the preset tolerance threshold (specifically set according to the required signal synchronization accuracy requirements, with the default setting being within 5% deviation from the reference axis).

[0094] In S6, the surgical instrument type switching event and the pathological feature mutation state are detected, and a dynamic priority allocation matrix of the modal tactile signal after the scene change is generated.

[0095] Identify the real-time image of the doctor's virtual reality device, detect the surgical instrument type switching event and the sudden change of pathological characteristics, generate the scene change trigger signal, and synchronize the surgical stage coding accordingly.

[0096] When a scene change trigger signal appears, the historical pathology database is queried to extract the surgical stage of the corresponding disease and the tactile signal diagnostic sensitivity index, and the multidimensional joint state space is reconstructed.

[0097] A sliding time window mechanism is used to dynamically collect time series data of each modal tactile signal after scene changes. Based on the time series modal tactile signal data after scene changes, the multi-dimensional reward value of the effectiveness of tactile information is recalculated (consistent with the steps in S2). Based on the deep deterministic policy gradient algorithm, with the joint state space as input and the QoS level allocation as the action space, the reward value is backpropagated to the modal transmission strategy, and the dynamic priority allocation matrix of the modal tactile signal is regenerated as the basis for adjusting the bandwidth strategy of each modal signal transmission.

[0098] Example 2

[0099] The difference between Example 2 of the present invention and Example 1 is that this example introduces a remote medical interaction system and method based on virtual reality.

[0100] Figure 2 The present invention provides a structural diagram of a virtual reality-based telemedicine interaction system, which includes a delay prediction module, a tactile signal priority allocation module, a bandwidth strategy update module, a signal compensation module, and an event switching module:

[0101] Delay prediction module: By fusing the joint features of modal tactile data and signal transmission status, it establishes a modal transmission delay prediction model for pressure, temperature, and vibration modes, and outputs the transmission delay prediction value of each modal tactile signal in real time;

[0102] The tactile signal priority allocation module recognizes the real-time image of the doctor's virtual reality device, obtains the type of surgical instrument and the pathological status of the virtual operating area, and simultaneously establishes a two-branch regression network to output a multi-dimensional reward value for the effectiveness of the tactile information. This reward value is back-propagated to the modal transmission strategy to generate a dynamic priority allocation matrix for the modal tactile signal.

[0103] Bandwidth Policy Update Module: Based on the modal tactile signal priority matrix, the SDN controller divides virtual channels and allocates initial bandwidth. It constructs a network state matrix and uses it as input to optimize bandwidth slicing strategies to minimize transmission latency. It then iteratively updates the Q value, monitoring latency changes for the highest-priority signal and rolling back the strategy if latency increases.

[0104] Signal compensation module: This module establishes a unified time coordinate system using the pressure signal as the reference axis, calculates the time offset of the temperature and vibration signals, and compensates them using cubic spline interpolation and finite impulse response filtering, respectively. Finally, the alignment effect is verified based on time domain correlation and spatial deviation.

[0105] Event switching module: detects surgical instrument type switching events and pathological feature mutation states. If a scene change trigger signal appears, the tactile signal priority allocation module generates a dynamic priority allocation matrix for the modal tactile signal after the scene change.

[0106] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0107] The above embodiments can be implemented in whole or in part via software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in the embodiments of this application are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0108] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0111] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0113] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0114] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0115] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A remote medical interaction method based on virtual reality, characterized in that: The steps include: S1: Synchronously collect multimodal tactile data and corresponding transmission state parameters of the patient-side device, and establish a joint feature vector of the fused modal tactile data and signal transmission state; S2: Establish a multimodal transmission delay prediction model based on the joint feature vector and output the transmission delay prediction value of each modality tactile signal in real time; S3: Identify the real-time image of the doctor's virtual reality device, extract the surgical stage of the disease corresponding to the image and the diagnostic sensitivity index of each modal tactile signal, and simultaneously establish a two-branch regression network to obtain a multi-dimensional reward value for the effectiveness of the tactile information. The reward value is back-propagated to the modal transmission strategy to generate a dynamic priority allocation matrix for the modal tactile signals. S4: Based on the priority sorting of the dynamic priority allocation matrix of the modal tactile signals, the modal virtual channels are divided. Based on the real-time network status of the virtual channels and the real-time transmission delay prediction value, the bandwidth slicing strategy is updated through Q value iteration with the goal of minimizing the transmission delay of the modal tactile signals. S5: Establish a cross-modal unified time coordinate system, calculate the time offset of each modal tactile signal relative to the reference axis and perform signal compensation; S6: Detect surgical instrument type switching events and pathological feature mutation states, and generate a dynamic priority allocation matrix for modal tactile signals after scene changes; In S2, a multimodal transmission delay prediction model is established based on the joint feature vector, and the transmission delay prediction value of each modal tactile signal is output in real time. Specifically, the following are included: The joint feature vector is input into a fully connected neural network. Through end-to-end supervised learning, a modal transmission delay prediction model for pressure, temperature, and vibration modes is established. The loss function adopts a weighted combination of root mean square error and variance of the predicted value. The real-time delay observation values ​​of each modal tactile signal transmission are input to dynamically adjust the modal transmission delay prediction model until the model prediction accuracy reaches the set index; Based on the adjusted model, the transmission delay prediction value of each modality tactile signal is output in real time; In S3, the real-time image of the doctor's virtual reality device is identified, and the surgical stage of the disease corresponding to the image and the diagnostic sensitivity index of each modal tactile signal are extracted. At the same time, a two-branch regression network is established to obtain a multi-dimensional reward value for the effectiveness of tactile information. The reward value is back-propagated to the modal transmission strategy to generate a dynamic priority allocation matrix for modal tactile signals. Specifically, it includes: Identify the real-time image of the doctor's virtual reality device, obtain the type of surgical instrument and the pathological status of the virtual operating area, and simultaneously search the historical pathology database to extract the surgical stage of the corresponding disease and the tactile signal diagnostic sensitivity index; The surgical stage is encoded as a discrete state variable, and the pathological state is mapped to a high-dimensional feature vector. These are input into the tensor fusion layer together with the signal sensitivity index to generate a multi-dimensional joint state space representation of the reinforcement learning model. A dual-branch regression network was established. The main branch received real-time tactile feedback scores from the doctor, while the auxiliary branch analyzed the frequency of pauses during surgery. The two branches jointly output a multi-dimensional reward value for the effectiveness of the tactile information. Based on the deep deterministic policy gradient algorithm, with the joint state space as input and the QoS level assignment as the action space, the reward value is back-propagated to the modal transmission strategy to generate a dynamic priority assignment matrix for the modal tactile signal; In S4, based on the priority sorting of the dynamic priority allocation matrix of the modal tactile signals, modal virtual channels are divided. Based on the real-time network status of the virtual channels and the real-time transmission delay prediction value, the bandwidth slicing strategy is updated through Q value iteration to minimize the transmission delay of the modal tactile signals. Specifically, the following steps are involved: Based on the dynamic priority allocation matrix of modal tactile signals, the SDN controller divides independent virtual channels, establishes a mapping relationship between tactile modalities and virtual channels, and allocates initial bandwidth share according to priority order; Collect the real-time throughput, packet loss rate, and transmission delay prediction values ​​of each virtual channel to build a network status matrix containing spatiotemporal characteristics; Taking the network state matrix as input and the bandwidth allocation action as output, the bandwidth slicing strategy is updated through Q value iteration with the goal of minimizing the transmission delay of the modal tactile signal. Monitor whether the transmission delay of the modal tactile signal with the highest priority increases after the bandwidth slice is updated. If so, trigger a policy rollback.

2. A virtual reality-based telemedicine interaction method according to claim 1, characterized in that: In S1, the multimodal tactile data and corresponding transmission state parameters of the patient-side device are synchronously collected to establish a joint feature vector of the fused modal tactile data and signal transmission state. Specifically, the following steps are involved: Multi-threaded synchronous acquisition of multimodal tactile data from each tactile sensor on the patient's device, as well as the transmission state parameters of the corresponding modal tactile signals, generates a multimodal tactile signal raw data set with a unified timestamp. The multimodal tactile signals include pressure, temperature, and vibration tactile signals. The time domain features of multimodal tactile signals are extracted, and the long short-term memory network is used to extract the parameter features of the transmitted signal transmission state, and a joint feature vector of the fused modal tactile data and the signal transmission state is established.

3. A virtual reality-based telemedicine interaction method according to claim 1, characterized in that: In S5, a cross-modal unified time coordinate system is established, the time offset of each modal tactile signal relative to the reference axis is calculated, and signal compensation is performed, specifically including: The pressure signal timestamp is selected as the reference axis to establish a cross-modal unified time coordinate system; Based on the reference axis time coordinate system, the timestamp sequences of the temperature and vibration signals are analyzed respectively, and the time offset of each modal tactile signal relative to the reference axis is calculated by linear regression method. In view of the temperature signal delay, a cubic spline interpolation algorithm is used in the receiving end buffer to generate a continuous signal sequence that is spatially and temporally aligned with the pressure signal using the reference axis time as the independent variable. According to the time offset of the vibration signal, a finite impulse response digital filter is established to perform phase correction on the vibration signal. The time domain correlation coefficient and spatial coordinate deviation of the compensated temperature, vibration signal and pressure signal are calculated, and the compensation is terminated when both indicators reach the preset tolerance threshold.

4. The virtual reality-based telemedicine interaction method according to claim 1, characterized in that: In S6, the surgical instrument type switching event and the pathological feature mutation state are detected, and the dynamic priority allocation matrix of the modal tactile signal after the scene change is generated specifically includes: Identify the real-time image of the doctor's virtual reality device, detect surgical instrument type switching events and pathological feature mutation states, and generate scene change trigger signals; When a scene change trigger signal appears, the historical pathology database is queried to extract the surgical stage of the corresponding disease and the tactile signal diagnostic sensitivity index, and the multidimensional joint state space is reconstructed; A sliding time window mechanism is used to dynamically collect time series data of tactile signals of each modality after scene changes; Based on the temporal modal tactile signal data after the scene change, the multi-dimensional reward value of the tactile information effectiveness is recalculated, and a dynamic priority allocation matrix of the modal tactile signals after the scene change is generated.

5. A virtual reality-based telemedicine interaction system, used to implement a virtual reality-based telemedicine interaction method according to any one of claims 1 to 4, characterized in that: Including delay prediction module, tactile signal priority allocation module, bandwidth strategy update module, signal compensation module, and event switching module: Delay prediction module: By fusing the joint features of modal tactile data and signal transmission status, it establishes a modal transmission delay prediction model for pressure, temperature, and vibration modes, and outputs the transmission delay prediction value of each modal tactile signal in real time; The tactile signal priority allocation module identifies the real-time image of the doctor's virtual reality device, obtains the type of surgical instrument and the pathological status of the virtual operating area, and simultaneously establishes a two-branch regression network to output a multi-dimensional reward value for the effectiveness of the tactile information. This reward value is back-propagated to the modal transmission strategy to generate a dynamic priority allocation matrix for the modal tactile signal. Bandwidth policy update module: Based on the modal tactile signal priority matrix, the SDN controller divides virtual channels and allocates initial bandwidth, constructs a network status matrix, and uses the matrix as input to optimize the bandwidth slicing strategy to minimize transmission delay. It also monitors the delay changes of the highest priority signal through iterative updates of the Q value and rolls back the strategy if the delay increases. Signal compensation module: This module establishes a unified time coordinate system using the pressure signal as the reference axis, calculates the time offset of the temperature and vibration signals, and compensates them using cubic spline interpolation and finite impulse response filtering, respectively. Finally, the alignment effect is verified based on time domain correlation and spatial deviation. Event switching module: detects surgical instrument type switching events and pathological feature mutation states. If a scene change trigger signal appears, the tactile signal priority allocation module generates a dynamic priority allocation matrix for the modal tactile signal after the scene change.

Images