Interactive processing method and system for multimedia data

Abstract

The invention discloses an interactive processing method and system for multimedia data, and belongs to the technical field of multimedia data processing. The method comprises the steps of performing hierarchical processing on multimedia image data to be interacted, and generating three-level image data including thumbnail layer data, browsing layer data and fine layer data; a user operation behavior of the master control end is detected in real time, a current data transmission mode is determined according to the type of the user operation behavior, and data of a corresponding level is selected from the three-level image data and transmitted to the slave end; the user operation behavior is converted into an incremental operation instruction, and the incremental operation instruction is synchronously pushed to a slave end through a low-delay channel; and the slave side reconstructs a display state consistent with that of the master control side locally according to the received hierarchical data and the incremental operation instruction. According to the invention, low-delay and high-quality multimedia data interactive transmission is realized under the condition of limited network bandwidth through self-adaptive layered transmission and an incremental instruction synchronization mechanism.

Description

Technical Field

[0001] This invention belongs to the field of multimedia data processing technology, and specifically relates to a method and system for interactive processing of multimedia data. Background Technology

[0002] With the rapid development of applications such as telemedicine, online education, and remote collaboration, the demand for real-time interactive sharing of multimedia data is increasing. Current technologies typically employ full-volume transmission or fixed-bitrate streaming for remote multimedia data interaction. These methods are prone to stuttering and latency issues when network bandwidth is limited or fluctuating, severely impacting user experience. Particularly in applications requiring high image quality, such as remote medical imaging consultations, traditional video streaming methods struggle to simultaneously meet the dual demands of low-latency response and high-quality display. Furthermore, existing screen-sharing technologies often transmit the host's actions synchronously with the screen, resulting in slave users passively receiving the screen and failing to achieve a truly synchronous interactive experience.

[0003] Therefore, how to achieve low-latency, high-quality interactive transmission of multimedia data under limited network bandwidth conditions has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The purpose of this invention is to provide a multimedia data interactive processing method and system. By performing layered processing on multimedia image data and adaptively selecting the transmission level according to user operation behavior, combined with the synchronous push mechanism of incremental operation instructions, low-latency, high-quality multimedia data interactive transmission under limited network bandwidth conditions is achieved, solving the technical problems of high latency, large bandwidth consumption, and poor user interaction experience caused by full transmission in the prior art.

[0005] In a first aspect, the present invention provides a method for interactive processing of multimedia data, the method comprising the following steps: Step S1: Perform layered processing on the multimedia image data to be interacted with to generate three-level image data, including thumbnail layer data, browsing layer data, and fine layer data.

[0006] Step S2: Real-time detection of user operation behavior on the master control terminal; determination of the current data transmission mode based on the type of user operation behavior; selection of the corresponding level of data transmission from the three-level image data to the slave terminal.

[0007] Step S3: Convert the user operation behavior into incremental operation instructions, and push the incremental operation instructions synchronously to the slave terminal through a low-latency channel.

[0008] Step S4: The slave terminal reconstructs a display state consistent with that of the master terminal locally based on the received hierarchical data and the incremental operation instruction.

[0009] Furthermore, the layered processing of the multimedia image data to be interacted with specifically includes: The original image sequence is downsampled by a first scaling factor to generate the thumbnail layer data, wherein the first scaling factor... The calculation formula is: ;in, This is the first downsampling level. The value is determined based on the original image resolution. and target scaled resolution Sure: Here, ⌈ ⌉ represents the floor operation.

[0010] The original image sequence is downsampled by a second scaling factor to generate the browsing layer data, wherein the second scaling factor... The calculation formula is: ;in, It is the second downsampling level, and satisfies This makes the second scaling factor .

[0011] The original resolution data of the original image sequence is retained as the fine layer data.

[0012] Furthermore, determining the current transmission mode based on the type of user operation behavior specifically includes: real-time sampling of user operations and calculation of operation speed feature values. : in, The displacement is the distance the user moves when operating the input device within the sampling time interval, expressed in pixels. The operation time interval represents the time difference between two adjacent samples, in milliseconds.

[0013] Furthermore, when When this occurs, it is determined to be a fast sliding operation, and the current transmission mode is identified as the thumbnail layer transmission mode.

[0014] when When this occurs, it is determined to be a slow browsing operation, and the current transmission mode is determined to be the browsing layer transmission mode.

[0015] when And duration Exceeding the preset stay threshold ： Or, when a user operation is detected to be a labeled operation, the current transmission mode is determined to be the fine-grained transmission mode; where, For the fast sliding speed threshold, For slow browsing speed threshold, satisfy .

[0016] Furthermore, based on historical user behavior data and current operation trends, a predictive model is used to calculate the first [number]th [character / item] in the image sequence. The probability value of a frame being viewed ,in For frame index, , This represents the total number of frames in the image sequence.

[0017] Furthermore, preloading the image data based on the probability values ​​specifically includes: preloading the image data based on the probability values ​​that satisfy... Preload fine-layer data into frames; for probability values ​​that satisfy Preload browsing layer data for frames; for probability values ​​that satisfy The frames are either not preloaded or only the thumbnail layer data is preloaded; among them, The first probability threshold, The second probability threshold satisfies .

[0018] Furthermore, the prediction model employs a multi-factor weighted scoring model, and the probability value The calculation formula is: ;in, The Sigmoid normalization function is defined as follows: This is used to map the weighted summation result to the (0,1) interval; The number of characteristic factors; For the first The weight coefficients of each feature satisfy... ; For the first The first frame Each feature value.

[0019] Furthermore, the characteristic factors include: spatial proximity characteristics, relevance characteristics of the area of ​​interest, behavioral trend characteristics, and historical habit characteristics.

[0020] Spatial proximity features Indicates the first The spatial proximity of a frame to the current browsing position is calculated using the following formula: in, This is the index of the currently viewed frame; This is the spatial attenuation coefficient, which controls the attenuation rate of proximity characteristics. .

[0021] Focus on regional correlation characteristics Indicates the first The importance of regions of interest identified by the pre-detection algorithm in a frame is calculated using the following formula: ;in, For the first A set of regions of interest identified in a frame using a pre-detection algorithm; For the first Confidence scores for each region of interest. ;when When it is an empty set, .

[0022] Behavioral trend characteristics This indicates that the target frame that the user may visit is predicted based on the current swipe direction and swipe speed. The calculation formula is as follows: ;in, Based on the current sliding direction and sliding speed The predicted target frame index is calculated as follows: ,in, Indicates sliding backward. Indicates sliding forward. For the prediction time window; This is the trend decay coefficient. ; Let be the direction indicator function, when the first... When the frame is in the prediction direction ,otherwise .

[0023] Historical Habits and Characteristics This indicates the user's browsing history and tendency to access the i-th frame. The calculation formula is as follows: ;in, For the first The number of historical accesses to a frame; This represents the total number of frames in the image sequence; when all frames have not been accessed... .

[0024] Furthermore, the data structure of incremental operation instructions includes: a global synchronization timestamp, an operation type identifier, and a set of operation parameters; The global synchronization timestamp represents the moment when an operation occurs based on a unified time base, in milliseconds; the operation type identifier includes at least one of frame switching operation, scaling operation, panning operation, window width and window level adjustment operation, and annotation operation; The set of operation parameters includes at least one of the following based on the operation type: image frame index, scaling factor and scaling center coordinates, translation offset, window width parameter and window level parameter, annotation type and annotation content.

[0025] Furthermore, real-time monitoring of current network bandwidth. and network latency Calculate the estimated transmission time for data at each level: ; in, These correspond to the thumbnail layer, browsing layer, and fine layer, respectively. For the first The size of the hierarchical data, in bits; For the first The estimated transmission time at each level; when the estimated transmission time exceeds the interaction response threshold. In such cases, the transmission will automatically downgrade to a lower-level layer. ;in, The actual transport layer used; interaction response threshold. This indicates the maximum acceptable waiting time for the user; When network bandwidth is restored and meets the requirements When necessary, it automatically upgrades to a higher level of transmission, among which This is the bandwidth recovery threshold.

[0026] Secondly, based on the same inventive concept, the present invention provides an interactive processing system for multimedia data, the system comprising: an image layering processing module, an adaptive transmission control module, an incremental instruction encoding module, and a slave state reconstruction module.

[0027] Furthermore, the image layering processing module is used to perform layered processing on the multimedia image data to be interacted with, generating multi-level image data including thumbnail layer data, browsing layer data, and fine layer data.

[0028] Furthermore, the adaptive transmission control module is used to detect user operation behavior at the master control end in real time, determine the current transmission mode according to the type of user operation behavior, select data of the corresponding level from the multi-level image data according to the transmission mode, and control the data transmission to the slave end.

[0029] Furthermore, the incremental instruction encoding module is used to convert the user operation behavior into incremental operation instructions and synchronously push the incremental operation instructions to the slave terminal.

[0030] Furthermore, the slave state reconstruction module is set on the slave end and is used to reconstruct the display state that is consistent with the master end locally based on the received hierarchical data and the incremental operation instruction.

[0031] Compared with the prior art, the beneficial effects of this invention are: This invention performs three-level layered processing on multimedia image data: thumbnail layer, browsing layer, and fine layer. It adaptively selects the corresponding layer of data for transmission based on the type of user operation behavior. When scrolling quickly, low-resolution thumbnail layer data is transmitted to ensure response speed. When browsing slowly, medium-resolution browsing layer data is transmitted to balance quality and speed. When observing or annotating, high-resolution fine layer data is transmitted to ensure display quality. This achieves a dynamic balance between low latency and high quality under limited network bandwidth conditions. Attached Figure Description

[0032] Figure 1 This is a flowchart of a multimedia data interactive processing method according to the present invention; Figure 2 This is a schematic diagram of the composition of a multimedia data interactive processing system according to the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0034] The technical solution of this invention will be described in detail below using remote medical image consultation as a specific application scenario. In this application scenario, the master control terminal is the workstation of the attending physician, the slave terminal is the terminal device of the consulting expert, and the multimedia image data to be interacted with is the patient's CT or MRI image sequence. The attending physician browses and annotates the images through the master control terminal, and the consulting expert watches the attending physician's operation process in real time through the slave terminal, realizing remote collaborative diagnosis.

[0035] Example 1 like Figure 1 As shown, this invention provides a method for interactive processing of multimedia data, the method comprising the following steps: Step S1: Perform layered processing on the multimedia image data to be interacted with to generate three-level image data, including thumbnail layer data, browsing layer data, and fine layer data.

[0036] The multimedia video data to be interacted with is processed in layers, specifically including: The original image sequence is downsampled by a first scaling factor to generate the thumbnail layer data, wherein the first scaling factor... The calculation formula is: ;in, This is the first downsampling level. The value is determined based on the original image resolution. and target scaled resolution Sure: Here, ⌈ ⌉ represents the floor operation.

[0037] The original image sequence is downsampled by a second scaling factor to generate the browsing layer data, wherein the second scaling factor... The calculation formula is: ;in, It is the second downsampling level, and satisfies This makes the second scaling factor The original resolution data of the original image sequence is retained as the fine layer data.

[0038] For example, for a sequence of 500 CT images, assuming the original image resolution is 512×512 pixels and the target thumbnail resolution is set to 64×64 pixels, then the first downsampling level n1 = ⌈log2(512 / 64)⌉ = 3, the first scaling factor S1 = 1 / 8, and the generated thumbnail layer data is only 64×64 pixels per frame, with a data volume of approximately 1 / 64 of the original data. If the target browsing resolution is set to 256×256 pixels, then the second downsampling level n2 = 1, the second scaling factor S2 = 1 / 2, and the generated browsing layer data is 256×256 pixels per frame, with a data volume of approximately 1 / 4 of the original data. Through the above layered processing, the same image sequence is preprocessed into three versions with different resolutions, laying the data foundation for subsequent adaptive selection of the transmission layer based on user operation behavior.

[0039] Step S2: Real-time detection of user operation behavior on the master control terminal; determination of the current data transmission mode based on the type of user operation behavior; selection of the corresponding level of data transmission from the three-level image data to the slave terminal.

[0040] Determining the current transmission mode based on the type of user operation behavior specifically includes: real-time sampling of user operations and calculation of operation speed feature values. : in, The displacement is the distance the user moves when operating the input device within the sampling time interval, expressed in pixels. The operation time interval represents the time difference between two adjacent samples, in milliseconds.

[0041] when When this occurs, it is determined to be a fast sliding operation, and the current transmission mode is identified as the thumbnail layer transmission mode.

[0042] when When this occurs, it is determined to be a slow browsing operation, and the current transmission mode is determined to be the browsing layer transmission mode.

[0043] when And duration Exceeding the preset stay threshold ： Or, when a user operation is detected to be a labeled operation, the current transmission mode is determined to be the fine-grained transmission mode; where, For the fast sliding speed threshold, For slow browsing speed threshold, satisfy .

[0044] In the specific implementation, the fast scrolling speed threshold V_high can be set to 100 pixels / millisecond, the slow browsing speed threshold V_low can be set to 10 pixels / millisecond, and the dwell time threshold T_th can be set to 500 milliseconds. For example, when the attending physician uses the mouse wheel to quickly scroll through the entire CT sequence to locate the lesion, the system detects that the operation speed V_op exceeds 100 pixels / millisecond and automatically switches to the thumbnail layer transmission mode. At this time, the slave device displays the image at a low resolution of 64×64 pixels, but the frame switching response delay can be controlled within 50 milliseconds. When the physician slows down the scrolling speed to carefully examine a certain area, the system detects that the operation speed is between 10 and 100 pixels / millisecond and switches to the browsing layer transmission mode. The slave device displays the image at a medium resolution of 256×256 pixels. When the physician dwells on a certain frame image for more than 500 milliseconds to prepare for lesion annotation, the system automatically switches to the fine layer transmission mode. The slave device displays the image at a high resolution of the original 512×512 pixels, ensuring that the consulting experts can clearly observe the details of the lesion.

[0045] This embodiment also provides another method for determining the current data transmission mode based on the type of user operation behavior: based on historical data of user operation behavior and current operation trends, a prediction model is used to calculate the data transmission mode of the image sequence. The probability value of a frame being viewed ,in For frame index, , This represents the total number of frames in the image sequence.

[0046] Preloading image data based on the probability values ​​specifically includes: preloading image data based on probability values ​​that satisfy... Preload fine-layer data into frames; for probability values ​​that satisfy Preload browsing layer data for frames; for probability values ​​that satisfy The frames are either not preloaded or only the thumbnail layer data is preloaded; among them, The first probability threshold, The second probability threshold satisfies .

[0047] The prediction model employs a multi-factor weighted scoring model, and the probability value... The calculation formula is: ;in, The Sigmoid normalization function is defined as follows: This is used to map the weighted summation result to the (0,1) interval; The number of characteristic factors; For the first The weight coefficients of each feature satisfy... ; For the first The first frame Each feature value.

[0048] The characteristic factors include: spatial proximity characteristics, correlation characteristics of the area of ​​interest, behavioral trend characteristics, and historical habit characteristics.

[0049] Spatial proximity features Indicates the first The spatial proximity of a frame to the current browsing position is calculated using the following formula: in, This is the index of the currently viewed frame; This is the spatial attenuation coefficient, which controls the attenuation rate of proximity characteristics. .

[0050] Focus on regional correlation characteristics Indicates the first The importance of regions of interest identified by the pre-detection algorithm in a frame is calculated using the following formula: ;in, For the first A set of regions of interest identified in a frame using a pre-detection algorithm; For the first Confidence scores for each region of interest. ;when When it is an empty set, .

[0051] Behavioral trend characteristics This indicates that the target frame that the user may visit is predicted based on the current swipe direction and swipe speed. The calculation formula is as follows: ;in, Based on the current sliding direction and sliding speed The predicted target frame index is calculated as follows: ,in, Indicates sliding backward. Indicates sliding forward. For the prediction time window; This is the trend decay coefficient. ; Let be the direction indicator function, when the first... When the frame is in the prediction direction ,otherwise .

[0052] Historical Habits and Characteristics This indicates the user's browsing history and tendency to access the i-th frame. The calculation formula is as follows: ;in, For the first The number of historical accesses to a frame; This represents the total number of frames in the image sequence; when all frames have not been accessed... .

[0053] The weighting coefficient Dynamic adjustments are made through the following adaptive learning mechanism: Initialize all weight coefficients to equal values: In each actual user access frame Then, the prediction accuracy index is calculated. : ;in, This represents the predicted probability value of the actual access frame to the user before the access. A higher value indicates a more accurate prediction.

[0054] Update the weighting coefficients based on the contribution of each feature to the prediction accuracy: ;in, The learning rate parameter, ; To update the number of iterations; The first frame actually accessed by the user There are several eigenvalues; when the prediction is inaccurate, i.e. When smaller, For features that contribute significantly more, increase their weight. The updated weights are then normalized. .

[0055] Using the aforementioned predictive model, the system can intelligently predict which image frames a doctor will be viewing and preload the corresponding layer data. For example, when a doctor is viewing frame 100 and scrolling backward, the system assigns higher probability values ​​to frames 101 to 110 based on spatial proximity characteristics, higher probability values ​​to frames containing pre-detected suspected lesion areas based on region of interest correlation characteristics, higher probability values ​​to frames near the predicted target frame based on behavioral trend characteristics, and higher probability values ​​to specific layers frequently visited by the doctor in past consultations based on historical habit characteristics. By combining the weighted scores of these features, the system can preload the fine-grained layer data of the target frame into the slave cache before the doctor actually switches to it, thereby achieving near-zero latency frame switching display.

[0056] Step S3: Convert the user operation behavior into incremental operation instructions, and push the incremental operation instructions synchronously to the slave terminal through a low-latency channel.

[0057] The data structure of incremental operation instructions includes: a global synchronization timestamp, an operation type identifier, and a set of operation parameters; The global synchronization timestamp represents the moment when an operation occurs based on a unified time base, in milliseconds; the operation type identifier includes at least one of frame switching operation, scaling operation, panning operation, window width and window level adjustment operation, and annotation operation; The set of operation parameters includes at least one of the following based on the operation type: image frame index, scaling factor and scaling center coordinates, translation offset, window width parameter and window level parameter, annotation type and annotation content.

[0058] The method further includes: establishing a unified global time reference, synchronizing the clocks of each end through a network time protocol, and controlling the clock deviation of each end within a threshold range; inserting synchronization anchors in the video stream, audio stream, and operation command stream at preset time intervals; and associating and storing the synchronization anchors with the status of each media stream at the corresponding time to form a time index table to support the synchronous backtracking of the multimedia timeline.

[0059] For example, when the attending physician performs the following operations, the system generates corresponding incremental operation instructions: When the physician switches the image from frame 100 to frame 105, a frame switching instruction is generated, containing the target frame index 105; when the physician zooms in on the image to 200% magnification and positions the center of the field of view at coordinates (256, 300), a scaling instruction is generated, containing a scaling factor of 2.0 and the scaling center coordinates (256, 300); when the physician adjusts the window width and window level to optimize the display effect of soft tissue, a window width and window level adjustment instruction is generated, containing a new window width value of 350 and a window level value of 50; when the physician uses the circle tool to mark a suspected lesion area on the image, a marking instruction is generated, containing the marking type "circle", the center coordinates (280, 310), and a radius of 15 pixels. The data volume of the above incremental operation instructions is usually only a few tens of bytes, which can save more than 99% of bandwidth compared to transmitting complete image frame data.

[0060] Step S4: The slave terminal reconstructs a display state consistent with that of the master terminal locally based on the received hierarchical data and the incremental operation instruction.

[0061] Real-time monitoring of current network bandwidth and network latency Calculate the estimated transmission time for data at each level: ; in, These correspond to the thumbnail layer, browsing layer, and fine layer, respectively. For the first The size of the hierarchical data, in bits; For the first The estimated transmission time at each level; when the estimated transmission time exceeds the interaction response threshold. In such cases, the transmission will automatically downgrade to a lower-level layer. ;in, The actual transport layer used; interaction response threshold. This indicates the maximum acceptable waiting time for the user; When network bandwidth is restored and meets the requirements When necessary, it automatically upgrades to a higher level of transmission, among which This is the bandwidth recovery threshold.

[0062] In practical implementation, the interaction response threshold T_resp is typically set to 200 milliseconds to ensure that users do not perceive any noticeable operation delay. For example, when the network bandwidth is 10 Mbps and the network latency is 20 milliseconds, transmitting a frame of 512×512 pixel fine-layer data (approximately 256KB) takes approximately 226 milliseconds, exceeding the 200-millisecond response threshold. The system automatically downgrades to transmitting browsing-layer data (approximately 64KB), with a transmission time of approximately 71 milliseconds, meeting the response requirements. If the network deteriorates further, the system can continue to downgrade to transmitting thumbnail-layer data (approximately 4KB), with a transmission time of only 23 milliseconds. When the network stabilizes and the bandwidth exceeds the bandwidth recovery threshold B_th (e.g., 20 Mbps), the system automatically upgrades the transmission layer, gradually restoring high-quality display.

[0063] Example 2 like Figure 2 The diagram shown is a schematic representation of the composition of an interactive multimedia data processing system according to the present invention. The system includes: an image layering processing module, an adaptive transmission control module, an incremental instruction encoding module, and a slave state reconstruction module.

[0064] The four functional modules can be flexibly configured on different physical or virtual nodes. The image layering processing module is usually deployed on a server with strong computing power to complete the preprocessing and storage of three-level data when the image data is uploaded for the first time. The adaptive transmission control module and the incremental instruction encoding module are deployed on the master control end to respond to user operations in real time and make transmission strategy decisions. The slave end state reconstruction module is deployed on the slave end to receive data and instructions and reconstruct the display state.

[0065] The modules exchange data through an efficient network communication protocol. The image data channel uses a reliable transmission protocol to ensure data integrity, while the incremental operation command channel uses a low-latency protocol to ensure real-time synchronization of operations.

[0066] The image layering processing module is used to perform layered processing on the multimedia image data to be interacted with, generating multi-level image data including thumbnail layer data, browsing layer data, and fine layer data.

[0067] The adaptive transmission control module is used to detect user operation behavior at the master end in real time, determine the current transmission mode according to the type of user operation behavior, select data of the corresponding level from the multi-level image data according to the transmission mode, and control the data transmission to the slave end.

[0068] The incremental instruction encoding module is used to convert the user operation behavior into incremental operation instructions, and synchronously push the incremental operation instructions to the slave terminal.

[0069] The slave state reconstruction module is set on the slave end and is used to reconstruct the display state that is consistent with the master end locally based on the received hierarchical data and the incremental operation instructions.

[0070] Furthermore, the image layering processing module includes an image decoding submodule, a multi-level downsampling submodule, and a layered storage submodule, which are responsible for decoding the original image data, generating data at each layer, and organizing and storing the layered data, respectively. The adaptive transmission control module includes an operation behavior acquisition submodule, an operation type determination submodule, a prediction preloading submodule, and a transmission scheduling submodule, which are responsible for acquiring user operation signals, determining the operation behavior type in real time, predicting the target frame intelligently, and scheduling and executing transmission tasks, respectively. The incremental instruction encoding module includes an operation parsing submodule, an instruction encoding submodule, and a synchronization push submodule, which are responsible for converting user operations into structured data, encoding instructions according to a unified format, and pushing them to the slave end through a low-latency channel, respectively. The slave end state reconstruction module includes a data receiving submodule, an instruction parsing submodule, a cache management submodule, and a rendering and display submodule, which are responsible for receiving layered data and operation instructions, parsing instruction content, managing multi-level data caches, and rendering and displaying the screen according to the current state, respectively.

[0071] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for interactive processing of multimedia data, characterized in that, The method includes the following steps: Step S1: Perform layered processing on the multimedia image data to be interacted with to generate three-level image data, including thumbnail layer data, browsing layer data, and fine layer data. Step S2: Real-time detection of user operation behavior on the master control terminal; determination of the current data transmission mode based on the type of user operation behavior; selection of the corresponding level of data transmission from the three-level image data to the slave terminal. Step S3: Convert the user operation behavior into incremental operation instructions, and push the incremental operation instructions synchronously to the slave terminal through a low-latency channel; Step S4: The slave terminal reconstructs a display state consistent with that of the master terminal locally based on the received hierarchical data and the incremental operation instruction.

2. The method according to claim 1, characterized in that, The layered processing of the multimedia image data to be interacted with specifically includes: The original image sequence is downsampled by a first scaling factor to generate the thumbnail layer data, wherein the first scaling factor... The calculation formula is: ;in, This is the first downsampling level. The value is determined based on the original image resolution. and target scaled resolution Sure: Where ⌈ ⌉ represents the floor function; The original image sequence is downsampled by a second scaling factor to generate the browsing layer data, wherein the second scaling factor... The calculation formula is: ;in, It is the second downsampling level, and satisfies This makes the second scaling factor ; The original resolution data of the original image sequence is retained as the fine layer data.

3. The method according to claim 2, characterized in that, The step of determining the current transmission mode based on the type of user operation behavior specifically includes: real-time sampling of user operations and calculation of operation speed feature values. : in, The displacement is the distance the user moves when operating the input device within the sampling time interval, expressed in pixels. The operation time interval represents the time difference between two adjacent samples, in milliseconds.

4. The method according to claim 3, characterized in that, when When this is detected, it is determined to be a fast sliding operation, and the current transmission mode is determined to be the thumbnail layer transmission mode; when When this occurs, it is determined to be a slow browsing operation, and the current transmission mode is set to browsing layer transmission mode. when And duration Exceeding the preset stay threshold ： Alternatively, if a user operation is detected as a labeled operation, the current transmission mode is determined to be the fine-grained transmission mode. in, For the fast sliding speed threshold, For slow browsing speed threshold, meet .

5. The method according to claim 4, characterized in that, Based on historical user behavior data and current operation trends, a predictive model is used to calculate the first [number]th [character / item] in the image sequence. The probability value of a frame being viewed. ,in For frame index, , This represents the total number of frames in the image sequence. Preloading the image data based on the probability values ​​specifically includes: For probability values ​​satisfy Frame preloading of fine-grained layer data; For probability values ​​satisfy Frame preload browsing layer data; For probability values ​​satisfy The frames are not preloaded or only the thumbnail layer data is preloaded; in, The first probability threshold, The second probability threshold satisfies .

6. The method according to claim 5, characterized in that, The prediction model employs a multi-factor weighted scoring model, and the probability value... The calculation formula is: ;in, The Sigmoid normalization function is defined as follows: This is used to map the weighted summation result to the (0,1) interval; The number of characteristic factors; For the first The weight coefficients of each feature satisfy... ; For the first The first frame Each feature value.

7. The method according to claim 6, characterized in that, The characteristic factors include: spatial proximity characteristics, relevance characteristics of the area of ​​interest, behavioral trend characteristics, and historical habit characteristics; Spatial proximity features Indicates the first The spatial proximity of a frame to the current browsing position is calculated using the following formula: in, This is the index of the currently viewed frame; This is the spatial attenuation coefficient, which controls the attenuation rate of proximity characteristics. ; Focus on regional correlation characteristics Indicates the first The importance of regions of interest identified by the pre-detection algorithm in a frame is calculated using the following formula: ;in, For the first A set of regions of interest identified in a frame using a pre-detection algorithm; For the first Confidence scores for each region of interest. ;when When it is an empty set, ; Behavioral trend characteristics This indicates that the target frame that the user may visit is predicted based on the current swipe direction and swipe speed. The calculation formula is as follows: ;in, Based on the current sliding direction and sliding speed The predicted target frame index is calculated as follows: ,in, Indicates sliding backward. Indicates sliding forward. For the prediction time window; This is the trend decay coefficient. ; Let be the direction indicator function, when the first... When the frame is in the prediction direction ,otherwise ; Historical Habits and Characteristics This indicates the user's browsing history and tendency to access the i-th frame. The calculation formula is as follows: ;in, For the first The historical number of times a frame has been accessed; This represents the total number of frames in the image sequence; when all frames have not been accessed... .

8. The method according to claim 7, characterized in that, The data structure of incremental operation instructions includes: a global synchronization timestamp, an operation type identifier, and a set of operation parameters; The global synchronization timestamp represents the moment when an operation occurs based on a unified time base, in milliseconds; the operation type identifier includes at least one of frame switching operation, scaling operation, panning operation, window width and window level adjustment operation, and annotation operation; The set of operation parameters includes at least one of the following based on the operation type: image frame index, scaling factor and scaling center coordinates, translation offset, window width parameter and window level parameter, annotation type and annotation content.

9. The method according to claim 8, characterized in that, Real-time monitoring of current network bandwidth and network latency Calculate the estimated transmission time for data at each level: ; in, These correspond to the thumbnail layer, browsing layer, and fine layer, respectively. For the first The size of the hierarchical data, in bits; For the first The estimated transmission time at each level; when the estimated transmission time exceeds the interaction response threshold. In such cases, the transmission will automatically downgrade to a lower-level layer. ;in, The actual transport layer used; interaction response threshold. This indicates the maximum acceptable waiting time for the user; When network bandwidth is restored and meets the requirements When necessary, it automatically upgrades to a higher level of transmission, among which This is the bandwidth recovery threshold.

10. A multimedia data interactive processing system for performing the method according to any one of claims 1-9, characterized in that, The system includes: an image layering processing module, an adaptive transmission control module, an incremental instruction encoding module, and a slave state reconstruction module; The image layering processing module is used to perform layered processing on the multimedia image data to be interacted with, generating multi-level image data including thumbnail layer data, browsing layer data, and fine layer data. The adaptive transmission control module is used to detect user operation behavior at the master end in real time, determine the current transmission mode according to the type of user operation behavior, select data of the corresponding level from the multi-level image data according to the transmission mode, and control the data transmission to the slave end. The incremental instruction encoding module is used to convert the user operation behavior into incremental operation instructions, and synchronously push the incremental operation instructions to the slave terminal; The slave state reconstruction module is set on the slave end and is used to reconstruct the display state that is consistent with the master end locally based on the received hierarchical data and the incremental operation instructions.

Images