A real-time interactive method and system for multimodal rehabilitation medical information

Abstract

The present invention relates to the technical field related to rehabilitation information communication interaction, and specifically includes a real-time interaction method and system suitable for multimodal rehabilitation medical information, including: determining a multimodal data stream, generating an initial interaction network architecture, using a heterogeneous data feature encoder, adding up- and down-sampling feature pointers, and dynamically adapting elements to perform dynamic allocation optimization, using different interaction requests to perform short-term communication adjustments according to the dynamic adaptation elements, and performing configuration optimization according to the real-time scheduling of the multimodal data stream, setting an adaptive interaction network architecture, solving the technical problem that the feature extraction and fusion capabilities of multimodal data are insufficient under different interaction requests and data stream conditions, and the interaction requirements cannot be met, realizing the path selection optimization in the multi-path transmission topology structure, and performing short-term communication adjustments according to different interaction requests and data stream conditions, to ensure the technical effect that the interaction request can be quickly responded to and processed.

Description

Technical Field

[0001] The present invention relates to the technical field related to rehabilitation information communication interaction, and in particular to a real-time interaction method and system applicable to multimodal rehabilitation medical information. Background Art

[0002] Multimodal input and output aims to enable the system to simultaneously process data input from multiple different modalities, such as images, text, voice, etc., and can output results in a variety of modal forms accordingly, providing users with a richer, more intuitive and more demand-oriented interactive experience. At this stage, we have begun to gradually explore the use of rehabilitation robots to assist users in rehabilitation treatment. However, most of the current rehabilitation robots have problems such as lack of flexible controllability of the drive and a single training mode, which cannot meet the efficiency and safety requirements of rehabilitation treatment. At the same time, in the processing of medical information, conventional single-modal processing methods have gradually shown limitations when dealing with massive and complex medical data information, including medical images (such as CT, MRI, etc.), text records in electronic medical records, voice content of communication between doctors and users, and various physiological signals (such as electrocardiograms, electroencephalograms, etc.). Therefore, there is an urgent need to efficiently integrate and process multimodal medical data to improve the overall efficiency of medical work.

[0003] In summary, the prior art has the technical problem that the feature extraction and fusion capabilities of multimodal data under different interaction requests and data streams are insufficient and cannot meet the interaction requirements. Summary of the invention

[0004] The present application provides a real-time interactive system suitable for multimodal rehabilitation medical information, aiming to solve the technical problem in the prior art that the feature extraction and fusion capabilities of multimodal data are insufficient under different interaction requests and data flow conditions, and cannot meet the interaction needs.

[0005] In view of the above problems, the technical solution to implement this application is:

[0006] On the one hand, the present application provides a real-time interaction method for multimodal rehabilitation medical information, wherein the method comprises: performing timestamp alignment on rehabilitation medical information including physiological electrical signals and kinematic data uploaded by a wearable device and behavioral state information including voice interaction data and facial micro-expressions uploaded by a monitoring device to determine a multimodal data stream; generating an initial interaction network architecture based on the multimodal data stream, wherein the initial interaction network architecture comprises interaction node elements, interaction response elements, and dynamic adaptation elements; based on the initial interaction network architecture, using a heterogeneous data feature encoder, adding upsampling feature pointers and downsampling feature pointers, wherein the upsampling feature pointers comprise U index mapping chains, and the downsampling feature pointers comprise V index aggregation chains; performing dynamic allocation optimization with the dynamic adaptation elements of the initial interaction network architecture according to the upsampling feature pointers and the downsampling feature pointers, wherein the dynamic adaptation elements comprise path switching dynamic adaptation elements and path stabilization dynamic adaptation elements; in the initial interaction network architecture, using different interaction requests according to the dynamic adaptation elements to perform short-term communication adjustment, and performing configuration optimization according to the real-time scheduling of the multimodal data stream, and setting an adaptive interaction network architecture.

[0007] On the other hand, the present application provides a real-time interactive system suitable for multimodal rehabilitation medical information, wherein the system includes: a timestamp alignment module for performing timestamp alignment on rehabilitation medical information including physiological electrical signals and kinematic data uploaded by wearable devices and behavioral state information including voice interaction data and facial micro-expressions uploaded by monitoring devices to determine a multimodal data stream; an architecture generation module for generating an initial interactive network architecture based on the multimodal data stream, wherein the initial interactive network architecture includes interactive node elements, interactive response elements, and dynamic adaptation elements; a pointer addition module for adding the acquired points based on the initial interactive network architecture using a heterogeneous data feature encoder. The upsampling feature pointer and the downsampling feature pointer include U index mapping chains and the downsampling feature pointer includes V index aggregation chains; an allocation optimization module is used to perform dynamic allocation optimization with the dynamic adaptation elements of the initial interactive network architecture according to the upsampling feature pointer and the downsampling feature pointer, and the dynamic adaptation elements include path switching dynamic adaptation elements and path stabilization dynamic adaptation elements; a communication adjustment module is used to perform short-term communication adjustment using different interaction requests according to the dynamic adaptation elements in the initial interactive network architecture, and perform configuration optimization according to the real-time scheduling of the multimodal data stream to set an adaptive interactive network architecture.

[0008] In summary, one or more technical solutions provided in the present application dynamically adjust path connections according to real-time transmission status and interaction requirements, efficiently process real-time interaction of multimodal rehabilitation medical information, and utilize upsampling feature pointers and downsampling feature pointers to automatically adjust the configuration according to different interaction requirements and data flow conditions, thereby achieving path selection optimization in a multi-path transmission topology structure, and performing short-term communication adjustments according to different interaction requests and data flow conditions, thereby ensuring the technical effect that interaction requests can be quickly responded to and processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flowchart of a real-time interactive method for multimodal rehabilitation medical information is provided for this application;

[0010] Figure 2 A structural schematic diagram of a real-time interactive system suitable for multimodal rehabilitation medical information is provided for this application.

[0011] Description of the reference numerals: timestamp alignment module 100 , architecture generation module 200 , pointer addition module 300 , allocation optimization module 400 , communication adjustment module 500 . DETAILED DESCRIPTION

[0012] Embodiment 1

[0013] The present application is described in detail below with reference to the accompanying drawings. Figure 1 As shown, the present application provides a real-time interaction method applicable to multimodal rehabilitation medical information, wherein the method comprises:

[0014] S1: Align the timestamps of the rehabilitation medical information uploaded by the wearable device, including physiological electrical signals and kinematic data, and the behavioral status information uploaded by the monitoring device, including voice interaction data and facial micro-expressions, to determine the multimodal data stream; S2: Based on the multimodal data stream, generate an initial interactive network architecture, which includes interactive node elements, interactive response elements, and dynamic adaptation elements.

[0015] Specifically, timestamp alignment refers to the synchronous processing of data from different devices (such as physiological electrical signals from wearable devices and voice interaction data from monitoring devices) in chronological order. Since there is a certain probability that the clocks of different devices will deviate, timestamp alignment can ensure the consistency of data in the time dimension, providing a basis for subsequent data processing and analysis; multimodal data stream refers to the integration of data from multiple modalities (such as physiological signals, kinematic data, voice and facial expressions, etc.) into a unified data stream for comprehensive processing and analysis; the initial interaction network architecture is used to manage and optimize data interaction, including interaction node elements (nodes for data interaction), interaction response elements (response mechanisms to interaction requests) and dynamic adaptation elements (the ability to adjust network configuration according to real-time needs).

[0016] The multimodal data from wearable devices and monitoring devices are timestamped. The physiological electrical signals and kinematic data uploaded by the wearable devices, as well as the voice interaction data and facial micro-expression data uploaded by the monitoring devices, need to be timestamped to eliminate time deviations due to device clock differences and data acquisition time asynchronization. Furthermore, by introducing synchronous pulse signals or time correction algorithms, the consistency of multimodal data in the time dimension is ensured, thereby forming a unified multimodal data stream.

[0017] Based on the aligned multimodal data stream, the initial interactive network architecture is generated. The initial interactive network architecture provides a basic framework for the interaction of multimodal data by defining interactive node elements, interactive response elements and dynamic adaptation elements. The interactive node elements are responsible for receiving and forwarding data, the interactive response elements are used to process interactive requests and generate responses, and the dynamic adaptation elements dynamically adjust the network configuration according to real-time data streams and interactive requests. The role of the initial interactive network architecture is to provide a basis for subsequent dynamic optimization and adaptive interaction, ensuring that the system can flexibly respond to different interactive needs and achieve efficient processing and interaction of multimodal rehabilitation medical information.

[0018] S3: Based on the initial interactive network architecture, a heterogeneous data feature encoder is used to add upsampling feature pointers and downsampling feature pointers, wherein the upsampling feature pointers include U index mapping chains and the downsampling feature pointers include V index aggregation chains; S4: According to the upsampling feature pointers and downsampling feature pointers, dynamic allocation optimization is performed with the dynamic adaptation elements of the initial interactive network architecture, and the dynamic adaptation elements include path switching dynamic adaptation elements and path stabilization dynamic adaptation elements; S5: In the initial interactive network architecture, according to the dynamic adaptation elements, different interactive requests are used to perform short-term communication adjustments, and configuration optimization is performed according to the real-time scheduling of the multimodal data stream to set an adaptive interactive network architecture.

[0019] Specifically, the heterogeneous data feature encoder is used to process multimodal data, and can encode data from different sources and formats (such as physiological electrical signals, voice data, kinematic data, etc.) into a unified feature representation for subsequent processing and analysis; the upsampling feature pointer is used to upsample the data features to increase the resolution or dimension of the features to extract richer detail information; the downsampling feature pointer is used to downsample the data features to reduce feature redundancy and improve processing efficiency, and the index mapping chain and index aggregation chain are used for path planning for upsampling and downsampling respectively; the dynamic adaptation element is a module used to dynamically adjust the network configuration in the initial interactive network architecture, including the path switching dynamic adaptation element (used to quickly switch paths to adapt to different interactive requests) and the path stability dynamic adaptation element (used to maintain the stability of the path and ensure the continuity of data transmission); short-term communication adjustment refers to the rapid adjustment of the communication path and resource allocation in the network architecture according to the real-time interaction request and data flow status to meet the current interaction needs; the adaptive interactive network architecture is a dynamically optimized network architecture that can automatically adjust the configuration according to the real-time multimodal data flow and interaction request to achieve efficient and flexible interaction.

[0020] A heterogeneous data feature encoder is used to extract and optimize the features of multimodal data. Specifically, upsampling and downsampling operations are performed on the data features by adding upsampling feature pointers and downsampling feature pointers. The upsampling feature pointer contains U index mapping chains, which are used to map low-resolution or low-dimensional features to high-resolution or high-dimensional features, thereby extracting richer detail information; the downsampling feature pointer contains V index aggregation chains, which are used to aggregate high-resolution or high-dimensional features to low-resolution or low-dimensional features, thereby reducing redundancy and improving processing efficiency.

[0021] Based on the upsampling and downsampling feature pointers, the initial interaction network architecture is dynamically allocated and optimized using dynamic adaptation elements. The dynamic adaptation elements include path switching dynamic adaptation elements and path stabilization dynamic adaptation elements. The path switching dynamic adaptation elements are used to quickly switch communication paths according to real-time interaction requests to adapt to different interaction needs; the path stabilization dynamic adaptation elements are used to maintain the stability of the path to ensure the continuity and reliability of data transmission.

[0022] In the initial interactive network architecture, different interactive requests are used for short-term communication adjustments based on the optimization results of the dynamic adaptation elements, and configuration optimization is performed according to the real-time scheduling of multimodal data streams. Furthermore, by dynamically adjusting the communication paths and resource allocation in the network architecture, it is ensured that interactive requests can be responded to quickly and processed efficiently, forming an adaptive interactive network architecture. The adaptive interactive network architecture can automatically adjust the configuration according to real-time multimodal data streams and interactive requests, thereby realizing real-time, efficient and intelligent interaction of multimodal rehabilitation medical information.

[0023] Furthermore, the rehabilitation medical information including physiological electrical signals and kinematic data uploaded by the wearable device and the behavioral state information including voice interaction data and facial micro-expressions uploaded by the monitoring device are time-stamp aligned. The method of the present application also includes:

[0024] Through the synchronization pulse signal, the first relative drift rate between the main node and the wearable device and the second relative drift rate between the main node and the monitoring device are determined; based on the first relative drift rate and the second relative drift rate, the time mapping parameters are set, and the time mapping parameters are subjected to residual analysis to determine whether there is nonlinear drift; if there is no nonlinear drift, hierarchical data cache management is performed.

[0025] Specifically, a synchronization pulse signal is a signal used for time synchronization. By sending and receiving pulse signals between different devices, the time deviation and relative drift rate between devices can be determined; the relative drift rate refers to the drift speed of the clocks of two devices relative to each other, that is, the difference between the clock frequencies of the two devices; time mapping parameters are parameters used to map the time of different devices to a unified time standard, usually including time offset and time scaling factor; residual analysis is a method of determining whether there is nonlinear drift by calculating the residual of the time mapping parameters (that is, the difference between the actual value and the predicted value); hierarchical data cache management is a data management strategy that caches and manages data according to different levels (such as time sequence, data type, etc.) to improve data access efficiency and reduce latency.

[0026] By sending and receiving synchronization pulse signals, the first relative drift rate between the master node and the wearable device, as well as the second relative drift rate between the master node and the monitoring device are determined, which is crucial to ensuring the time synchronization of multimodal data, because there is a certain probability that the clocks of different devices have frequency differences, resulting in time drift; based on the determined relative drift rate, the time mapping parameters are set to map the time of different devices to a unified time standard. Further, the time mapping parameters are subjected to residual analysis, that is, the difference between the actual time mapping result and the ideal time mapping result is calculated to determine whether there is nonlinear drift. Nonlinear drift refers to the nonlinear trend of the time mapping parameters changing over time, which is generally caused by the nonlinear characteristics of the device clock or external environmental factors.

[0027] If the residual analysis shows that there is no nonlinear drift, that is, the residual of the time mapping parameter is within an acceptable range, then hierarchical data cache management can be performed. Hierarchical data cache management caches and manages multimodal data according to time sequence or data type, thereby improving data access efficiency, reducing processing delays, and ensuring the continuity and real-time nature of multimodal data streams.

[0028] Furthermore, the present application method also includes:

[0029] If nonlinear drift exists, a polling scheduling mechanism is set, and M sliding windows are inserted, where M≥3; based on the M sliding windows, the mutual information index corresponding to the multimodal data stream is evaluated; when the mutual information index is lower than a preset index threshold, a retransmission signal is triggered.

[0030] Specifically, nonlinear drift refers to the nonlinear change of device clock drift, that is, the drift rate changes over time and is unstable, which is difficult to solve by simple linear correction methods; the polling scheduling mechanism refers to periodically checking and adjusting system resource allocation to ensure that each device or task can obtain processing opportunities in a predetermined order and time interval; the sliding window refers to moving a fixed-size window in the data stream to analyze and process the data in the window. The sliding window can be used to monitor the characteristic changes of the data stream in real time; the mutual information index is an indicator to measure the correlation and information sharing between data streams, and is used to evaluate the quality and consistency of multimodal data streams; the retransmission signal is used to retransmit data to ensure the integrity and accuracy of the data.

[0031] When the residual analysis indicates the existence of nonlinear drift, more complex strategies are needed to deal with the time synchronization problem of multimodal data streams. At this time, a polling scheduling mechanism is set up and M sliding windows (where M ≥ 3) are inserted into the data stream. The polling scheduling mechanism periodically checks the status of the data stream to ensure that the system can respond to changes caused by nonlinear drift in a timely manner. The introduction of sliding windows is used to monitor the characteristic changes of the data stream in real time. By moving the window in the data stream, the data in the window is analyzed.

[0032] Based on the M sliding windows, the mutual information index of the multimodal data stream is evaluated. The mutual information index reflects the correlation and information sharing degree between different modal data, and is an important indicator for measuring the quality and consistency of data streams. If the mutual information index is lower than the preset threshold, it means that there are large errors or lost information in the data stream, and the requirements of real-time interaction cannot be met. At this time, a retransmission signal is triggered, requiring related devices to retransmit data to ensure the integrity and accuracy of the data; through a dynamic monitoring and adjustment mechanism, the data synchronization problem caused by nonlinear drift is solved to ensure the reliability and consistency of multimodal data streams under complex conditions; through a combination of polling scheduling and sliding windows, the state of the data stream can be detected in real time, and errors can be corrected through a retransmission mechanism when necessary.

[0033] Furthermore, the present application method also includes:

[0034] Define time mapping parameters , the residual of the ith data point ; Dynamic correction factor corresponding to time drift ;in, The time derivative is used to evaluate the rate of change of the time mapping parameter error over time, is the timestamp of the ith data point, is the time offset of the ith data point, is the output value of the time mapping parameter corresponding to the i-th data point, is the total number of data points, is the learning rate, which is used to update the dynamic correction factor.

[0035] Specifically, the time mapping parameter A set of parameters used to map the time bases of different devices to a unified time standard, usually including a time offset and a time scaling factor, used to correct the time differences between devices; the residual of the i-th data point It refers to the difference between the actual timestamp of the ith data point and the timestamp calculated by the time mapping parameter; the dynamic correction factor corresponding to the time drift ,Furthermore, the dynamic correction factor is an adjustable parameter ,used to correct the time drift in real time and is dynamically ,updated according to the change of the residual to reduce the ,time mapping error; is the time derivative, which is the rate at which the residual changes over time. It is used to evaluate the changing trend of the time mapping parameter error, reflects the dynamic characteristics of time drift, and is the basis for updating the dynamic correction factor; learning rate It is a hyperparameter used to control the speed of updating the dynamic correction factor. A higher learning rate will lead to a faster adjustment of the correction factor, but there is a certain probability of instability; a lower learning rate will make the adjustment process smoother, but the convergence speed will be slower.

[0036] Define and calculate time mapping parameters and their related parameters to achieve dynamic correction of time drift. Specifically, define the time mapping parameters , unify the time bases of different devices into a standard time frame, and for the i-th data point, calculate its residual, that is, the difference between the actual timestamp and the timestamp calculated by the time mapping parameters. The size of the residual directly reflects the accuracy of the time mapping; introduce the dynamic correction factor corresponding to the time drift Furthermore, the update speed of the dynamic correction factor is controlled by the learning rate. The learning rate determines the response speed of the correction factor to the residual change. Appropriate selection of the learning rate can ensure the rapid convergence of the correction factor while avoiding system instability caused by too fast adjustment; through the dynamic correction mechanism, the time mapping parameters are adjusted in real time to adapt to the time drift between devices. The dynamic correction factor and the learning rate are combined to correct the time mapping error in real time. When facing complex nonlinear drifts, the synchronization and consistency of the multimodal data stream in the time dimension are ensured, thereby ensuring the efficient processing and real-time interaction of multimodal rehabilitation medical information.

[0037] Furthermore, the upsampling feature pointer includes U index mapping chains, and the method of the present application includes:

[0038] Based on the U index mapping chains in the upsampled feature pointer, a first multi-path transmission topology is constructed to collect bandwidth utilization between each interaction node under different interaction requests; according to the bandwidth utilization between each interaction node under the first interaction request, a first interaction node set is generated, and according to different interaction requests, a second interaction node set, a third interaction node set, ..., a Pth interaction node set are obtained; based on the first interaction node set, the second interaction node set, the third interaction node set, ..., the Pth interaction node set, a multi-objective optimization algorithm is used to perform path allocation competition optimization to determine the interaction response element.

[0039] Specifically, the upsampling feature pointer is used to enhance the resolution or dimension of data features, while the index mapping chain is a specific link in the pointer used to define the feature mapping path. U index mapping chains indicate that there are multiple paths for feature upsampling processing; the multi-path transmission topology allows data to be transmitted through multiple paths to improve the reliability and efficiency of the system, and the paths can be dynamically adjusted to adapt to different network conditions and interaction requirements; bandwidth utilization refers to the ratio of the bandwidth actually used in the network link to the total available bandwidth, reflecting the utilization efficiency of network resources; the interactive node set refers to the classification of interactive nodes into different sets according to different interaction requests and bandwidth utilization, and each set represents a group of nodes with similar characteristics or requirements; the multi-objective optimization algorithm is used to consider multiple objectives (such as bandwidth utilization, path delay, etc.) at the same time to find the optimal path allocation solution.

[0040] A multipath transmission topology is constructed based on the index mapping chain in the upsampled feature pointer, and different interaction requests are met by dynamically optimizing path allocation. Specifically, a first multipath transmission topology is constructed using U index mapping chains in the upsampled feature pointer. The first multipath transmission topology allows data to be transmitted through multiple paths, thereby improving the flexibility and reliability of the system. In multipath transmission, the bandwidth utilization between each interaction node under different interaction requests is collected to evaluate the usage of network resources and potential bottlenecks; a first interaction node set is generated based on the bandwidth utilization under the first interaction request, and a second interaction node set, a third interaction node set, and ... a Pth interaction node set are generated in turn based on different interaction requests, and each set represents a group of interaction nodes with similar characteristics or requirements.

[0041] Based on the first interaction node set, the second interaction node set, the third interaction node set, ..., the Pth interaction node set, in multi-path transmission, the optimal path is selected through a competition mechanism to meet different interaction requests and optimization goals. The multi-objective optimization algorithm dynamically adjusts the path allocation by considering multiple goals (such as maximizing bandwidth utilization, minimizing path delay, etc.) to meet different interaction requests, and then determines the optimal interaction response element, that is, the path selection scheme with the highest adaptability under the current network conditions; by dynamically optimizing path allocation, the efficiency and flexibility of the multi-path transmission topology structure are improved; by collecting bandwidth utilization and generating an interaction node set, the path allocation is dynamically adjusted according to the real-time network status and interaction needs to improve the efficiency of multimodal rehabilitation medicine information interaction.

[0042] Furthermore, the present application method also includes:

[0043] The first interaction node set includes a load-sharing interaction node subset corresponding to high bandwidth utilization, a balanced allocation interaction node subset corresponding to medium bandwidth utilization, and a dormant integration interaction node subset corresponding to low bandwidth utilization; based on the V index aggregation chains in the downsampling feature pointer, a second multi-path transmission topology structure is constructed, and according to the bandwidth utilization between each interaction node under the first interaction request, a P+1th interaction node set is generated; based on the first interaction node set and the P+1th interaction node set, bidirectional index items of the upsampling feature pointer and the downsampling feature pointer are set.

[0044] Specifically, load-sharing interaction node subsets refer to dividing interaction nodes into a subset under high bandwidth utilization to share network load and avoid single-point overload; balanced allocation interaction node subsets refer to dividing interaction nodes into a subset under medium bandwidth utilization to achieve balanced resource allocation in the network; dormant integration interaction node subsets refer to dividing interaction nodes into a subset under low bandwidth utilization to integrate resources and put some nodes into dormant state to save resources; downsampling feature pointers are used to reduce the dimension or resolution of data features, while index aggregation chains are specific links in pointers used to define feature aggregation paths, and V index aggregation chains indicate that there are multiple paths for feature downsampling processing; bidirectional index items refer to the index relationship established between upsampling and downsampling feature pointers, which are used to achieve efficient data transmission and optimized processing in multi-path transmission topology structures.

[0045] Based on the index aggregation chain in the first interaction node set and the downsampling feature pointer, the multipath transmission topology is further optimized, and a bidirectional index item is set between the upsampling and downsampling feature pointers to achieve more efficient resource allocation and data processing. Specifically, the first interaction node set is subdivided into a load diversion interaction node subset, a balanced allocation interaction node subset, and a dormant integration interaction node subset. Furthermore, the load diversion interaction node subset corresponding to high bandwidth utilization is used to reduce single-point pressure by diversion when the network load is high, thereby improving the overall performance of the system; the balanced allocation interaction node subset corresponding to medium bandwidth utilization is used to achieve balanced allocation of resources when the network load is medium, thereby ensuring load balance between nodes; the dormant integration interaction node subset corresponding to low bandwidth utilization is used to integrate resources and put some nodes into a dormant state when the network load is low, thereby saving resources and improving efficiency.

[0046] Based on the V index aggregation chains in the downsampling feature pointers, a second multi-path transmission topology is constructed. The second multi-path transmission topology aggregates the data features through the downsampling feature pointers to further optimize the data transmission path. At the same time, according to the bandwidth utilization between each interaction node under the first interaction request, the P+1th interaction node set is generated. The P+1th interaction node set reflects the network status and resource allocation after the downsampling processing.

[0047] Based on the first interaction node set and the P+1th interaction node set, bidirectional index items of upsampling feature pointers and downsampling feature pointers are set. The role of the bidirectional index items is to establish a mapping relationship between the upsampling and downsampling feature pointers, so that the system can dynamically adjust the data processing and transmission paths in the multi-path transmission topology to adapt to different interaction requests and network conditions; by subdividing the interaction node set and introducing bidirectional index items, the resource allocation and data processing efficiency of the multi-path transmission topology are further optimized; through load splitting, resource balanced allocation and sleep integration, the system can better adapt to different bandwidth utilization conditions; at the same time, the bidirectional index items are used to achieve efficient collaboration between the upsampling and downsampling feature pointers, thereby improving the overall performance and flexibility of the multimodal rehabilitation medicine information interaction system.

[0048] Furthermore, in the initial interactive network architecture, according to the dynamic adaptation element, different interactive requests are used to perform short-term communication adjustment, and the method of the present application includes:

[0049] In the initial interactive network architecture, different interactive requests are used for short-term communication to monitor the transmission status of the multimodal data stream; short-term communication adjustment is performed according to the transmission status of the multimodal data stream, and the mapping relationship of the bidirectional index items is updated to obtain the mapping relationship update record of the bidirectional index items; based on the mapping relationship update record of the bidirectional index items, adaptive optimization is performed using the path switching dynamic adaptation element and the path stabilization dynamic adaptation element in the dynamic adaptation element.

[0050] Specifically, short-term communication refers to the data interaction process completed in a relatively short period of time, which is usually used to quickly respond to interaction requests and ensure the real-time performance of the system; transmission status refers to the status information of multimodal data streams when they are transmitted in the network, including data integrity, delay, packet loss rate, bandwidth utilization, etc.; the mapping relationship update record of the bidirectional index item records the dynamic adjustment process of the mapping relationship between the upsampling feature pointer and the downsampling feature pointer during the short-term communication process; the path switching dynamic adaptation element is used to quickly switch paths in multi-path transmission to adapt to different interaction requests and network conditions; the path stability dynamic adaptation element is used to maintain the stability of the path to ensure the continuity and reliability of data transmission.

[0051] In the initial interactive network architecture, short-term communication is performed according to different interactive requests, and the transmission status of multimodal data streams is monitored in real time. The communication path and index mapping relationship are dynamically adjusted to achieve adaptive optimization of the system. Specifically, in the initial interactive network architecture, different interactive requests are used for short-term communication. The interactive requests come from users, medical staff or the automatic triggering mechanism within the system. During the short-term communication process, the transmission status of multimodal data streams is monitored in real time, including key indicators such as data integrity, delay, packet loss rate and bandwidth utilization, which reflects the data transmission quality and efficiency under the current network environment.

[0052] Short-term communication adjustments are made according to the monitored transmission status. For example, if it is detected that the delay of a certain path is too high or the packet loss rate increases, the communication path is dynamically adjusted and a better path is selected for data transmission. At the same time, the mapping relationship of the bidirectional index items is updated to reflect the data processing and transmission path changes after the path adjustment. Each update of the mapping relationship of the bidirectional index items is saved to obtain an update record of the mapping relationship of the bidirectional index items; based on the mapping relationship update record of the bidirectional index items, adaptive optimization is performed using the path switching dynamic adaptation element and the path stabilization dynamic adaptation element in the dynamic adaptation element to obtain an adaptive interactive network architecture. Furthermore, the path switching dynamic adaptation element is responsible for quickly switching the path to adapt to new interaction requests or changes in network status; the path stabilization dynamic adaptation element ensures the stability of the path during the switching process to avoid transmission interruption or data loss caused by frequent switching.

[0053] Through real-time monitoring and dynamic adjustment, the efficient transmission and stability of multimodal data streams in short-term communications are ensured. Through the optimization of dynamic adaptation elements, it can flexibly respond to different interaction requests and changes in network conditions, thereby improving the overall performance of the system and user experience, maintaining stable data transmission under high load or network fluctuations, and quickly responding to interaction requests according to real-time needs, effectively supporting real-time and efficient interaction of multimodal rehabilitation medical information, and improving the continuity and reliability of interactive communications.

[0054] In summary, the beneficial effects of the embodiments of the present application are:

[0055] By aligning the timestamps of rehabilitation medical information and behavioral status information, determining the multimodal data flow, generating the initial interactive network architecture, using heterogeneous data feature encoders, adding upsampling feature pointers and downsampling feature pointers, and dynamically allocating and optimizing with the dynamic adaptation elements of the initial interactive network architecture, in the initial interactive network architecture, according to the dynamic adaptation elements, short-term communication adjustments are made using different interactive requests, and configuration optimization is performed according to the real-time scheduling of the multimodal data flow, and an adaptive interactive network architecture is set. This application provides a real-time interaction method and system suitable for multimodal rehabilitation medical information, dynamically adjusts path connections according to real-time transmission status and interaction requirements, efficiently processes real-time interaction of multimodal rehabilitation medical information, uses upsampling feature pointers and downsampling feature pointers, automatically adjusts the configuration according to different interaction requirements and data flow conditions, realizes path selection optimization in a multi-path transmission topology, and performs short-term communication adjustments according to different interaction requests and data flow conditions, ensuring that interaction requests can be quickly responded to and processed. Technical effect.

[0056] Embodiment 2

[0057] Based on the same inventive concept as the real-time interactive method applicable to multimodal rehabilitation medical information in the aforementioned embodiment, Figure 2 As shown, the embodiment of the present application provides a real-time interactive system suitable for multimodal rehabilitation medical information, wherein the system includes:

[0058] The timestamp alignment module 100 is used to perform timestamp alignment on the rehabilitation medical information including physiological electrical signals and kinematic data uploaded by the wearable device and the behavioral state information including voice interaction data and facial micro-expressions uploaded by the monitoring device to determine the multimodal data stream.

[0059] The architecture generation module 200 is used to generate an initial interactive network architecture based on the multimodal data stream, wherein the initial interactive network architecture includes interactive node elements, interactive response elements, and dynamic adaptation elements.

[0060] The pointer adding module 300 is used to add upsampling feature pointers and downsampling feature pointers based on the initial interactive network architecture using a heterogeneous data feature encoder, wherein the upsampling feature pointer includes U index mapping chains and the downsampling feature pointer includes V index aggregation chains.

[0061] The allocation optimization module 400 is used to perform dynamic allocation optimization based on the upsampling feature pointer and the downsampling feature pointer using the dynamic adaptation elements of the initial interactive network architecture, wherein the dynamic adaptation elements include path switching dynamic adaptation elements and path stabilization dynamic adaptation elements.

[0062] The communication adjustment module 500 is used to adjust the short-term communication in the initial interactive network architecture according to the dynamic adaptation element using different interactive requests, and to optimize the configuration according to the real-time scheduling of the multimodal data flow, and to set the adaptive interactive network architecture.

[0063] Furthermore, the timestamp alignment module 100 is also used to perform the following method:

[0064] Through the synchronization pulse signal, the first relative drift rate between the master node and the wearable device and the second relative drift rate between the master node and the monitoring device are determined.

[0065] A time mapping parameter is set based on the first relative drift rate and the second relative drift rate, and a residual analysis is performed on the time mapping parameter to determine whether there is nonlinear drift.

[0066] If there is no nonlinear drift, hierarchical data cache management is performed.

[0067] Furthermore, the timestamp alignment module 100 is also used to perform the following method:

[0068] If there is nonlinear drift, a polling scheduling mechanism is set up and M sliding windows are inserted, where M ≥ 3.

[0069] According to the M sliding windows, the mutual information index corresponding to the multimodal data stream is evaluated.

[0070] When the mutual information index is lower than a preset index threshold, a retransmission signal is triggered.

[0071] Furthermore, the timestamp alignment module 100 is also used to perform the following method:

[0072] Define time mapping parameters , the residual of the ith data point .

[0073] Dynamic correction factor for time drift .

[0074] in, The time derivative is used to evaluate the rate of change of the time mapping parameter error over time, is the timestamp of the ith data point, is the time offset of the ith data point, is the output value of the time mapping parameter corresponding to the i-th data point, is the total number of data points, is the learning rate, which is used to update the dynamic correction factor.

[0075] Furthermore, the pointer adding module 300 is used to execute the following method:

[0076] Based on the U index mapping chains in the up-sampled feature pointer, a first multi-path transmission topology structure is constructed to collect bandwidth utilization between various interaction nodes under different interaction requests.

[0077] A first interactive node set is generated according to bandwidth utilization between each interactive node under a first interactive request, and a second interactive node set, a third interactive node set, ..., a Pth interactive node set are obtained according to different interactive requests.

[0078] Based on the first interactive node set, the second interactive node set, the third interactive node set, ..., the Pth interactive node set, a multi-objective optimization algorithm is used to perform path allocation competition optimization to determine the interactive response element.

[0079] Furthermore, the pointer adding module 300 is also used to execute the following method:

[0080] The first interactive node set includes a load distribution interactive node subset corresponding to high bandwidth utilization, a balanced allocation interactive node subset corresponding to medium bandwidth utilization, and a dormant integration interactive node subset corresponding to low bandwidth utilization.

[0081] Based on the V index aggregation chains in the down-sampled feature pointers, a second multi-path transmission topology is constructed, and according to the bandwidth utilization between the various interactive nodes under the first interactive request, a P+1th interactive node set is generated.

[0082] Based on the first interaction node set and the P+1th interaction node set, bidirectional index items of the upsampling feature pointer and the downsampling feature pointer are set.

[0083] Furthermore, the communication adjustment module 500 is used to perform the following method:

[0084] In the initial interactive network architecture, different interactive requests are used for short-term communication to monitor the transmission status of the multimodal data stream.

[0085] A short-term communication adjustment is performed according to the transmission status of the multimodal data stream, and the mapping relationship of the bidirectional index items is updated to obtain an update record of the mapping relationship of the bidirectional index items.

[0086] The mapping relationship record is updated based on the bidirectional index item, and adaptive optimization is performed using the path switching dynamic adaptation element and the path stabilization dynamic adaptation element in the dynamic adaptation element.

[0087] In summary, any step can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor, and no unnecessary restrictions are made here.

[0088] Furthermore, the above-mentioned technical scheme only reflects the preferred technical scheme of the technical scheme of the embodiment of the present application. Some changes that may be made to some parts thereof by technicians in this technical field all reflect the novel principles of the embodiment of the present application. Obviously, technicians in this field can make various changes and modifications to the present application without departing from the scope of the present application.

Claims

1. A real-time interactive method for multimodal rehabilitation medical information, characterized in that: The method comprises: Align the timestamps of rehabilitation medical information uploaded by wearable devices, including physiological electrical signals and kinematic data, and behavioral status information uploaded by monitoring devices, including voice interaction data and facial micro-expressions, to determine multimodal data streams; Based on the multimodal data stream, generating an initial interactive network architecture, the initial interactive network architecture comprising interactive node elements, interactive response elements, and dynamic adaptation elements; Based on the initial interactive network architecture, a heterogeneous data feature encoder is used to add upsampling feature pointers and downsampling feature pointers, wherein the upsampling feature pointer includes U index mapping chains and the downsampling feature pointer includes V index aggregation chains; According to the up-sampling feature pointer and the down-sampling feature pointer, dynamic allocation optimization is performed with the dynamic adaptation elements of the initial interactive network architecture, wherein the dynamic adaptation elements include a path switching dynamic adaptation element and a path stabilization dynamic adaptation element; In the initial interactive network architecture, according to the dynamic adaptation element, different interactive requests are used to perform short-term communication adjustments, and configuration optimization is performed according to the real-time scheduling of the multimodal data stream to set an adaptive interactive network architecture.

2. A real-time interactive method for multimodal rehabilitation medical information as claimed in claim 1, characterized in that: The method further includes: aligning the timestamps of the rehabilitation medical information including the physiological electrical signals and kinematic data uploaded by the wearable device and the behavioral state information including the voice interaction data and the facial micro-expression uploaded by the monitoring device. Determine a first relative drift rate between the master node and the wearable device and a second relative drift rate between the master node and the monitoring device through a synchronization pulse signal; Based on the first relative drift rate and the second relative drift rate, a time mapping parameter is set, and a residual analysis is performed on the time mapping parameter to determine whether there is nonlinear drift; If there is no nonlinear drift, hierarchical data cache management is performed.

3. A real-time interactive method for multimodal rehabilitation medical information as claimed in claim 2, characterized in that: If there is nonlinear drift, set up a polling scheduling mechanism and insert M sliding windows, where M ≥ 3; Evaluate the mutual information index corresponding to the multimodal data stream according to the M sliding windows; When the mutual information index is lower than a preset index threshold, a retransmission signal is triggered.

4. A real-time interactive method for multimodal rehabilitation medical information as claimed in claim 2, characterized in that: Define time mapping parameters , the residual of the ith data point ; Dynamic correction factor for time drift ; in, The time derivative is used to evaluate the rate of change of the time mapping parameter error over time, is the timestamp of the ith data point, is the time offset of the ith data point, is the output value of the time mapping parameter corresponding to the i-th data point, is the total number of data points, is the learning rate, which is used to update the dynamic correction factor.

5. A real-time interactive method for multimodal rehabilitation medical information as claimed in claim 4, characterized in that: The up-sampling feature pointer includes U index mapping chains, and the method includes: Based on the U index mapping chains in the up-sampled feature pointer, a first multi-path transmission topology structure is constructed to collect bandwidth utilization between various interaction nodes under different interaction requests; Generate a first interactive node set according to bandwidth utilization between each interactive node under the first interactive request, and obtain a second interactive node set, a third interactive node set, ..., a Pth interactive node set according to different interactive requests; Based on the first interactive node set, the second interactive node set, the third interactive node set, ..., the Pth interactive node set, a multi-objective optimization algorithm is used to perform path allocation competition optimization to determine the interactive response element.

6. A real-time interactive method for multimodal rehabilitation medical information as claimed in claim 5, characterized in that: The first interactive node set includes a load distribution interactive node subset corresponding to high bandwidth utilization, a balanced allocation interactive node subset corresponding to medium bandwidth utilization, and a dormant integration interactive node subset corresponding to low bandwidth utilization; Based on the V index aggregation chains in the down-sampled feature pointer, a second multi-path transmission topology is constructed, and according to the bandwidth utilization between each interactive node under the first interactive request, a P+1th interactive node set is generated; Based on the first interaction node set and the P+1th interaction node set, bidirectional index items of the upsampling feature pointer and the downsampling feature pointer are set.

7. A real-time interactive method for multimodal rehabilitation medical information as claimed in claim 6, characterized in that: In the initial interactive network architecture, according to the dynamic adaptation element, different interactive requests are used to perform short-term communication adjustment, and the method includes: In the initial interactive network architecture, different interactive requests are used for short-term communication to monitor the transmission status of the multimodal data stream; Performing short-term communication adjustment according to the transmission state of the multimodal data stream, and updating the mapping relationship of the bidirectional index item to obtain an update record of the mapping relationship of the bidirectional index item; The mapping relationship record is updated based on the bidirectional index item, and adaptive optimization is performed using the path switching dynamic adaptation element and the path stabilization dynamic adaptation element in the dynamic adaptation element.

8. A real-time interactive system for multimodal rehabilitation medical information, characterized in that: A system for implementing a real-time interactive method for multimodal rehabilitation medical information as described in any one of claims 1 to 7, comprising: The timestamp alignment module is used to align the timestamps of the rehabilitation medical information uploaded by the wearable device, including physiological electrical signals and kinematic data, and the behavioral state information uploaded by the monitoring device, including voice interaction data and facial micro-expression, to determine the multimodal data stream; An architecture generation module, configured to generate an initial interactive network architecture based on the multimodal data stream, wherein the initial interactive network architecture includes interactive node elements, interactive response elements, and dynamic adaptation elements; A pointer adding module, used to add upsampling feature pointers and downsampling feature pointers based on the initial interactive network architecture and using a heterogeneous data feature encoder, wherein the upsampling feature pointer includes U index mapping chains and the downsampling feature pointer includes V index aggregation chains; An allocation optimization module, configured to perform dynamic allocation optimization with the dynamic adaptation elements of the initial interactive network architecture according to the upsampling feature pointer and the downsampling feature pointer, wherein the dynamic adaptation elements include a path switching dynamic adaptation element and a path stabilization dynamic adaptation element; The communication adjustment module is used to perform short-term communication adjustment using different interaction requests in the initial interactive network architecture according to the dynamic adaptation elements, and to perform configuration optimization according to the real-time scheduling of the multimodal data stream to set an adaptive interactive network architecture.

Images