Method and apparatus for synchronously playing multimedia data based on time index

Abstract

The application relates to the technical field of multimedia synchronous playing, and discloses a multimedia data synchronous playing method and device based on time indexing, which comprises the following steps: extracting exclusive logical identification to determine application scene characteristic information and regulating and controlling concurrent processing priority according to the application scene characteristic information; constructing a differentiated hardware rendering pipeline; separating files to obtain audio main data and visual discrete data; calculating mixed timestamp information to generate linear timestamp which is not affected by frame rate; calculating predicted load density by using a load density formula and evaluating power load distribution; executing a cache intervention strategy; obtaining audio timestamp by using a main clock source; calculating the absolute time difference between the linear timestamp and the audio timestamp to generate a time sequence checking result; and executing normal rendering, track compensation rendering or dimension reduction rendering according to the checking result. The application eliminates the dependence of visual rendering on frame rate fluctuation by using linear timestamp conversion, load prediction and spatial dimension coordinate reconstruction mechanism, solves the audio and picture dislocation problem under high load, and optimizes playing instantaneity.

Description

Technical Field

[0001] This invention relates to the field of multimedia synchronous playback technology, specifically to a method and apparatus for synchronous playback of multimedia data based on time indexing. Background Technology

[0002] In video recording scenarios such as teaching recordings and live office broadcasts, multimedia data, after discretization and simplification, breaks down continuous video streams into non-continuous visual data such as static images, cursor trajectory sequences, handwritten annotation sequences, micro-video clips, and external material clips, and matches them with coherent audio data. This type of multimedia file has advantages in storage and transmission efficiency, but it requires high precision in timing scheduling during playback.

[0003] Existing technologies typically use a preset video frame rate as a time base for playback scheduling when processing discrete multimedia data. Under conditions of fluctuating recording environments or limited hardware encoding performance, the temporal information of visual data is prone to cumulative deviations, causing discontinuous visual units to fail to achieve precise synchronization with the audio sampling sequence. Simultaneously, the rendering pipeline of general-purpose playback devices cannot dynamically allocate hardware computing power for different application scenarios. When handling differentiated workloads such as handwriting anti-aliasing or micro-video parsing, computational resource contention can easily arise, leading to screen refresh stagnation or stuttering during transitions. Furthermore, when rendering latency occurs, existing scheduling mechanisms lack coordinate reconstruction compensation logic, typically resulting in delayed visual updates and difficulty maintaining audio-visual consistency. Consequently, the advantages of the simplified discrete data cannot be translated into a stable playback experience. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and apparatus for synchronous playback of multimedia data based on time indexing. This solves the problems of lack of a unified timing alignment benchmark and forward-looking hardware resource intervention methods in existing technologies, which easily lead to timing deviations, rendering stutters, and delayed synchronization feedback during playback.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The first aspect of this invention provides a method for synchronous playback of multimedia data based on time indexing, comprising the following steps:

[0007] Extracting the unique logical identifier of a discretized multimedia file determines the application scenario characteristics of that file. Based on this, the allocation strategy of hardware resources is dynamically reconstructed at the underlying level according to the application scenario characteristics. By adjusting the concurrent processing priority of the graphics processor, a differentiated hardware rendering pipeline matching the current application scenario characteristics is constructed. Through this configuration, hardware computing power is pre-allocated between video decoding and anti-aliasing processing, reducing the risk of playback lag at the underlying hardware level.

[0008] Discretized multimedia files are separated into main audio data and discrete visual data, and the hybrid timestamp information bound to the discrete visual data is calculated. Specifically, the hour, minute, and second portions of the hybrid timestamp information are converted into basic millisecond values, and combined with the absolute presentation time offset mapped onto the underlying audio sampling sequence to generate a linear timestamp. The linear timestamp removes the visual timeline's dependence on the video frame rate and utilizes the physical stability of the audio sampling sequence to establish a unified physical time reference system that is unaffected by encoding fluctuations.

[0009] Based on a backward sliding prediction of the time axis using linear timestamps, the rendering pressure of multimedia data within the future pre-read time window is quantified using a load density formula to obtain the predicted load density. An adaptive caching intervention strategy is implemented based on the relationship between this density and a safety threshold. Under high load distribution conditions, spatial dimension scheduling is performed across time sequences, establishing high-priority buffer channels in memory and loading rendering shaders into the graphics processor cache for warm-up. Through this mechanism, spatial dimension data scheduling compensates for temporal dimension rendering lag.

[0010] During playback, this scheme enables the master audio data to run at its native sampling rate to establish the master clock source and obtain the current audio timestamp. By calculating the absolute time difference between the audio timestamp and the linear timestamp that serves as the target start and end timestamps, this difference is compared with a tolerance threshold and an interpolable compensation threshold for verification, generating a timing verification result.

[0011] Finally, the final rendering method for the visual discrete data is determined based on the timing verification results.

[0012] In addition to normal rendering, for situations where the rendering progress lags behind the master clock source, this solution calls the trajectory compensation formula to map the coordinate state that the current time node should have in the spatial dimension, generates the target spatial coordinate point, and pushes it to the rendering queue.

[0013] Alternatively, in environments with limited computing power, smooth interpolation can be abandoned, and the coordinates of the core nodes can be extracted to perform dimensionality reduction rendering, thereby keeping the rendering progress of the visual discrete data synchronized with the master clock source.

[0014] This mechanism uses spatial coordinate reconstruction to correct temporal lags and maintain audio-visual consistency.

[0015] A second aspect of the present invention provides a multimedia data synchronous playback device based on time indexing, including a data parsing module, a cache control module, an audio and video rendering module, a timing comparison module, and a playback scheduling module.

[0016] The data parsing module is used to extract unique logical identifiers to determine application scenario characteristics and to separate discretized multimedia files and generate and allocate linear timestamps.

[0017] The cache control module is used to perform sliding prediction based on linear timestamps, calculate the predicted load density using the load density formula, and execute the corresponding cache intervention strategy.

[0018] The audio and video rendering module contains an audio processing unit, which is used to establish the master clock source and obtain audio timestamps in real time.

[0019] The timing comparison module is used to calculate the absolute time difference between the audio timestamp and the linear timestamp, and compare the tolerance threshold with the interpolation compensation threshold to generate timing verification results.

[0020] The playback scheduling module is used to build differentiated hardware rendering pipelines and perform regular rendering, compensated rendering, or dimensionality reduction rendering based on timing verification results to complete synchronous playback.

[0021] This invention provides a method and apparatus for synchronized playback of multimedia data based on time indexing. It has the following beneficial effects:

[0022] 1. This invention utilizes a linear timestamp conversion mechanism to directly map the temporal attributes of discrete visual data to the absolute rendering time offset corresponding to the underlying audio sampling sequence, eliminating the dependence of visual rendering progress on video frame rate fluctuations. By establishing a physical time scale with the audio stream as the primary clock source, the audio-visual synchronization accuracy in variable frame rate environments is improved.

[0023] 2. This invention constructs differentiated hardware rendering pipelines by extracting unique logical identifiers and uses a load density formula to predict future rendering pressure. In high-load scenarios, it pre-executes rendering shader warm-up and establishes high-priority buffer channels to prevent data congestion and rendering stagnation caused by sudden hardware load spikes.

[0024] 3. This invention, by invoking a trajectory compensation formula and performing dimensionality reduction rendering under specific conditions, compensates for temporal lag by reconstructing spatial coordinates or mapping core nodes when rendering delays occur in visual data. This solution, while ensuring uninterrupted audio stream output, returns the visual rendering trajectory to the master clock source, improving the immediacy of synchronized multimedia playback. Attached Figure Description

[0025] Figure 1 This is a flowchart of a multimedia data synchronization playback method based on time indexing according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of the spatiotemporal dual-drive adaptive cache scheduling process based on load density prediction according to an embodiment of the present invention.

[0027] Figure 3This is an architectural diagram of a multimedia data synchronization playback device based on time indexing according to an embodiment of the present invention;

[0028] Figure 4 This is a comparison chart of the synchronization effects based on time index in an application embodiment of the present invention. Detailed Implementation

[0029] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see the appendix Figure 1 This invention provides a method for synchronous playback of multimedia data based on time indexing, comprising the following steps:

[0031] S1. Method Initialization and Differentiated Hardware Rendering Pipeline Allocation Based on Scene Identifiers. This involves acquiring discretized multimedia files from external input, extracting their unique logical identifiers to determine the application scene characteristics. Based on these application scene characteristics, the underlying hardware resource allocation strategy is dynamically reconstructed, the concurrent processing priority of the graphics processor is adjusted, and a differentiated hardware rendering pipeline matching the application scene characteristics is constructed.

[0032] S2. Multi-source discrete data demultiplexing and frame-time absolute decoupling transformation. Discretized multimedia files are separated into audio master data and visual discrete data, and the mixed timestamp information bound to each type of visual discrete data is extracted. Numerical transformation and bias accumulation calculations are performed on the mixed timestamp information to generate linear timestamps unaffected by the video frame rate, and these linear timestamps are assigned to the corresponding visual discrete data as temporal attributes. This step aims to eliminate the visual timeline's dependence on variable frame rates, thereby establishing a unified absolute time reference system.

[0033] S3. Spatiotemporal Dual-Drive Adaptive Cache Scheduling Based on Load Density Prediction. Based on a backward sliding prediction of the time axis established by linear timestamps, the predicted load density within the future time window is calculated using the load density formula. According to the relationship between the predicted load density and a safety threshold, corresponding cache intervention strategies are executed: during low load distribution, a memory reclamation instruction is triggered to reclaim image memory; during high load distribution, a spatial dimension scheduling is switched to establish a high-priority buffer channel for visual discrete data and preload the rendering shaders into the graphics processor cache.

[0034] S4. Establish the audio master clock reference and absolute timing verification. Ensure the audio master data plays continuously at its native sampling rate, establishing the playback clock of the audio master data as the master clock source, and acquire the current audio timestamp in real time. Calculate the absolute time difference between the audio timestamp and the target start and end timestamps. Compare the absolute time difference with pre-set tolerance thresholds and interpolation compensation thresholds to generate corresponding timing verification results.

[0035] S5. Dynamic rendering scheduling and dimensionality reduction interpolation compensation based on timing verification results. The final rendering method of the visual discrete data is determined based on the timing verification results. When the timing verification result indicates normal rendering, the visual discrete data is pushed for normal rendering to complete normal synchronous playback; when the timing verification result indicates that interpolation compensation is required, the trajectory compensation formula is called to map the coordinate state in the spatial dimension, generate the target spatial coordinate points, and push them to the rendering queue; when the timing verification result indicates that dimensionality reduction rendering is required, the calculation of the trajectory compensation formula is abandoned to perform dimensionality reduction rendering, the coordinates of the core nodes are pushed, and dimensionality reduction synchronous playback is completed, thereby keeping the rendering progress of the visual discrete data synchronized with the master clock source.

[0036] The technical solutions in the embodiments of the present invention will be described in detail below:

[0037] In this embodiment, step S1 uses a data parsing module to parse the logical identifiers embedded in the multimedia file to determine the application scenario. Then, based on the scenario characteristics, it dynamically reconstructs the underlying hardware computing resources of the computer to build a differentiated hardware rendering pipeline adapted to the current scenario. Step S1 specifically includes:

[0038] S101. During the method initialization phase, obtain the discretized multimedia file input from the outside and extract the unique logical identifier of the discretized multimedia file.

[0039] In this embodiment, a unique logical identifier is typically encapsulated in the file header or a specific metadata segment of the discretized multimedia file, indicating the application scenario to which the file belongs at the time of its creation. Extracting this unique logical identifier allows determination of the application scenario characteristics of the current discretized multimedia file. For the underlying data operations of extracting metadata from the multimedia file and reading the file header identifier, those skilled in the art can employ conventional file parsing techniques, which will not be elaborated upon here.

[0040] S102. Based on the aforementioned application scenario characteristic information, dynamically reconstruct the hardware resource allocation strategy at the underlying level.

[0041] As a preferred approach, instructions are sent to the operating system kernel or underlying drivers to adjust the concurrent processing priority of the graphics processor, thereby constructing a differentiated hardware rendering pipeline that matches the characteristics of the application scenario. Adjusting the concurrent processing priority mainly involves reconfiguring the processing queues and weights of the stream processors, video decoding hardware acceleration modules (such as NVDEC or VCN hardware decoding units), and rendering output units within the graphics processor.

[0042] In actual execution, the extracted unique logical identifiers are matched with preset scene mapping rules. The preset scene mapping rules are mapping relationship tables pre-established based on the characteristics of audio and video data streams in different teaching or office scenarios.

[0043] If the dedicated logical identifier matches a high-frequency dynamic interaction type (e.g., the file contains identifiers like .exp or .demo representing experiments or demonstrations), then the application scenario characteristic information is confirmed to represent a high-frequency video interaction scenario. In this scenario, the screen frequently displays switching between micro-video clips or the insertion of external materials at high frame rates, placing high demands on the throughput of the video decoding hardware. In this case, a resource scheduling instruction is sent to the graphics processor driver to increase the allocation weight of the hardware video decoder thread, prioritizing the video memory throughput of the micro-video rendering unit. Specifically, increasing the allocation weight includes increasing the processor time slice length allocated to the decoder driver or increasing the response priority for video memory read requests in bus bandwidth scheduling.

[0044] Conversely, if the dedicated logical identifier matches a static display type (e.g., the file contains identifiers like .lesson or .meeting representing lectures or meetings), then the application scenario characteristic information confirms that it represents a low-frequency static whiteboard scenario. In this scenario, the background is mainly static graphics or static whiteboard writing that remains unchanged for a long time. The main visual updates come from the handwritten annotation sequence, requiring high smoothness and low latency in the handwriting display, while the computational demand for video decoding is relatively weak. At this time, by calling the dynamic power management interface or the application's underlying thread masking mechanism, some idle video decoding cores can be actively suspended, and the released computational resources can be concentrated on anti-aliasing processing (e.g., configured to a higher level of multi-sample anti-aliasing MSAA mode) and low-latency rendering channels for processing the handwritten annotation sequence, thereby completing the construction of a differentiated hardware rendering pipeline.

[0045] In addition, to ensure the robustness of system operation, when a dedicated logical identifier cannot be extracted, or when the dedicated logical identifier fails to match any preset scene type, a balanced allocation strategy will be adopted by default. That is, no bias weighting will be applied to specific hardware video decoder threads or anti-aliasing rendering channels. Instead, the default standard hardware pipeline of the operating system will be used for resource allocation to avoid rendering scheduling logic from getting stuck in a dead zone.

[0046] In this embodiment, step S2 utilizes a data parsing module to separate multi-source heterogeneous data encapsulated within the same file. Addressing the common frame rate fluctuation problem in multimedia file recording and transmission, it eliminates the dependence of the visual timeline on the video frame rate, thereby establishing a unified physical time reference system for each parallel data stream. Step S2 specifically includes:

[0047] S201. Separate the acquired discretized multimedia file into audio master data and visual discrete data including static images, cursor trajectory, handwriting, micro-videos, and external materials, and extract the mixed timestamp information bound to each type of visual discrete data.

[0048] In practice, by parsing the underlying container data packets of the discretized multimedia file, and based on the track header identifier or data block type bit of the data packet, a continuous audio encoded stream is independently extracted as the main audio data. After the audio stream extraction is completed, the accompanying video content is further classified and parsed into five independent categories of visual discrete data according to spatial and temporal characteristics, and the hybrid timestamp information associated with these data units is extracted simultaneously.

[0049] As a preferred method, the hybrid timestamp information is typically recorded in a standard timecode format that includes hours, minutes, seconds, and a tail frame offset. For the demultiplexing operation of reading the underlying data packets of a multimedia file and separating the data stream, those skilled in the art can use conventional streaming media parsing protocols (such as MP4 or FLV decapsulation mechanisms). The principles of data reading and track separation are well-known in the field and will not be elaborated upon here.

[0050] S202. To address the potential variable frame rate issue in multi-source data, extract the hour, minute, and second portions from the acquired mixed timestamp information, and multiply them by the corresponding millisecond conversion rate to calculate the base millisecond value that is unaffected by the video frame rate.

[0051] During the recording of multimedia files, the variable frame rate is limited by the performance fluctuations of the hardware encoder or resource scheduling. If the tail frame number in the timestamp is directly converted into physical time using a preset fixed video frame rate, the cumulative error will gradually increase as the playback time goes on, resulting in audio-visual misalignment.

[0052] To mitigate the impact of this hardware difference, a time estimation logic independent of the video frame rate is employed. In the specific calculation process, the hour, minute, and second values ​​are extracted from the mixed timestamp information and multiplied by the conversion constant from the higher-order time unit to milliseconds: hour value multiplied by 3,600,000, minute value multiplied by 60,000, and second value multiplied by 1,000. Summing these products yields a basic millisecond value independent of the video rendering mechanism.

[0053] Based on the aforementioned baseline millisecond value, the frame offset of less than one second or the absolute presentation time offset mapped to the underlying audio sampling sequence from the mixed timestamp information is further read. Since the audio sampling sequence is directly driven by a fixed audio sampling clock at the hardware level, its time distribution exhibits high linearity and stability.

[0054] In this embodiment, the specific method for determining the absolute presentation time offset is to divide the absolute index value of the sampling point of the target frame or event in the audio track by the native sampling rate of the audio master data (e.g., 44100Hz or 48000Hz), thereby accurately calculating the actual playback offset duration (in milliseconds) corresponding to the frame number offset node on the continuous audio sampling sequence, and using it as the absolute presentation time offset.

[0055] Add this absolute rendering time offset to the previously calculated base millisecond value to generate an absolute millisecond-level linear timestamp that is unaffected by the video frame rate. and the linear timestamp The corresponding discrete visual data is assigned as a temporal attribute. Through the aforementioned underlying transformation mechanism, the temporal attribute of the discrete visual data is accurately mapped onto a linear time axis based on an audio clock, establishing a unified judgment benchmark for implementing high-precision timing and caching interventions in subsequent steps.

[0056] See appendix Figure 2 In this embodiment, step S3 utilizes the cache control module to assess future computing load based on absolute timestamps, thereby guiding memory and video memory to schedule in advance, in order to reduce the risk of stuttering during multimedia data playback due to sudden high loads. Step S3 specifically includes:

[0057] S301, Based on the generated linear timestamp The established time-axis backward sliding prediction is used to calculate the multimedia data rendering pressure within the future time window using the load density formula, thus obtaining the predicted load density. .

[0058] In practice, a sliding detection window is deployed on a continuous timeline established by millisecond-level linear timestamps to extract the scale of various multimedia data within the coverage area of ​​the window.

[0059] Set pre-read time window duration It is set based on a combination of the storage bus read rate and the average transmission bit rate of multimedia files, such as the length of the prefetch window. It can be set from 2000 milliseconds to 5000 milliseconds to balance memory capacity limitations and the forward-looking needs of load prediction.

[0060] Based on this, the load density formula is used to output the quantitative evaluation results. The load density formula is as follows:

[0061] ;

[0062] In the formula: For linear timestamps Predicted load density at that time; The summation symbol; The total number of categories in the visual discrete data; To give the first based hardware rendering pipeline Weighting coefficients for computational power consumption of visual discrete data; To be within the set pre-read time window Inner The number of occurrences of visual discrete data or the volume of data to be processed; The set pre-read time window duration.

[0063] The computing power consumption weighting coefficient in the above formula It is not a fixed constant, but dynamically allocated based on the real-time configuration status of the differentiated hardware rendering pipeline. In this embodiment, the base rendering cost of static images is set to 1, and the initial weights of different data types are pre-defined (e.g., handwriting is set to 2, and micro-videos are set to 5).

[0064] When entering a specific application scenario, and a certain type of visual discrete data does not receive acceleration support from the hardware video decoder thread, a preset hardware missing penalty multiplier (e.g., 1.5 to 2.0 times) will be retrieved and multiplied by the corresponding initial weight, thereby increasing the computational cost weight coefficient of this type of data accordingly, so as to truly reflect the computational cost required to process the data.

[0065] S302, Obtain the generated predicted load density When judging the predicted load density When the load density is less than or equal to the safety threshold, confirm the predicted load density. For low-load distribution, a memory reclamation command is triggered to actively reclaim idle image memory, thus completing low-load cache intervention.

[0066] The safety threshold is set based on the maximum concurrent computing power of the current device's graphics processor and the available physical video memory. For example, the safety threshold can be set to 60% of the graphics processor's maximum throughput.

[0067] Under low-load distribution conditions, it is determined that data parsing and rendering requests in subsequent time periods will generally not put pressure on current hardware resources. As a preferred underlying implementation, by sending an instruction set to the display driver to execute memory reclamation instructions, texture cache objects or vertex buffer objects generated in the past playback and no longer in use are destroyed, thus actively reclaiming idle image memory. For the specific release instructions and memory addressing scheduling of the underlying memory buffer, those skilled in the art can use the conventional memory management mechanisms provided by various mainstream graphics application programming interfaces (such as OpenGL or Vulkan), and their memory release logic is well-known in the art and will not be elaborated here.

[0068] S303, When judging the predicted load density When the load density exceeds a preset safety threshold, confirm the predicted load density. To handle high load distribution, the system then switches to spatial dimension scheduling, prioritizing the establishment of high-priority buffer channels in memory for the upcoming visual discrete data, and pre-loading the rendering shaders corresponding to the visual discrete data into the graphics processor cache to warm them up, thus completing the high-load cache intervention.

[0069] Upon detecting a high load distribution, it is determined that subsequent playback is likely to encounter intensive data parsing or heavy image rendering. To avoid data congestion, the conventional method of reading frames sequentially by time is changed, and the aforementioned spatial dimension scheduling is implemented. This scheduling mechanism locks a contiguous physical address space in main memory based on the physical memory usage of the data to be processed, establishes a high-priority buffer channel without the risk of page table misses, and guides the corresponding data in external memory to be preferentially filled into this channel through direct memory access technology.

[0070] Simultaneously, a preheating process for the rendering shaders is initiated. Specifically, the underlying shading language code required for high-load data is pre-compiled to generate the corresponding binary executable instructions, and the rendering shaders corresponding to these visual discrete data are pre-loaded into the graphics processor cache. This hardware-level preheating operation helps reduce frame drops and stuttering caused by real-time shader code compilation during subsequent rendering, thereby improving the smoothness of overall multimedia data synchronization.

[0071] In this embodiment, step S4 involves the audio / video rendering module and the timing comparison module working together to establish a continuous and stable audio stream as the main time axis for advancement. It also monitors in real time whether the actual rendering progress of various types of discrete visual data is consistent with this audio axis, thus providing a basis for timing deviation in the subsequent synchronization compensation mechanism. Step S4 specifically includes:

[0072] S401. During the playback process, the audio processing unit inside the audio and video rendering module acquires the audio master data separated in step S201, and enables the audio master data to play continuously at the native sampling rate. The playback clock of the audio master data is established as the master clock source, which is used as the reference for subsequent visual discrete data synchronization and alignment, and the current audio timestamp is acquired in real time.

[0073] In multimedia data playback architecture, considering the human ear's sensitivity to sound stuttering and pitch shifts, the underlying audio driver interface (such as DirectSound or CoreAudio) directly drives the audio hardware digital-to-analog converter to decode and output at a uniform speed according to the native sampling rate of the main audio data (such as 44100Hz or 48000Hz for standard audio). Since this process is typically maintained by the crystal oscillator pulses of the hardware-level sound card, its playback progress exhibits high stability. This playback clock is established as the master clock source, and the total number of currently played samples is obtained by periodically polling the buffer position pointer of the underlying audio application programming interface.

[0074] As a further detail, the total number of played samples is divided by the native sampling rate, then multiplied by 1000 to obtain the audio timestamp in milliseconds. .

[0075] S402, The time-series comparison module obtains the linear timestamps assigned to the visual discrete data in step S202. Linear timestamp Calculate the audio timestamp as the target start and end timestamps. The absolute time difference between the target start and end timestamps .

[0076] Before the rendering engine executes the drawing, it needs to define the theoretical physical rendering nodes of each visual element. (Linear timestamp) This represents the theoretically expected nodes of the visual discrete data, which are directly mapped to the target start and end timestamps.

[0077] In the specific computational logic, the timing comparison module uses a hardware timer or a high-precision timer to extract the current audio timestamp. And calculate audio timestamps Start and end timestamps of the target The absolute value of the difference (i.e. This allows us to determine the absolute time difference between the current video progress and the baseline audio progress. This difference dynamically reflects the amount of rendering lag or lead produced by the graphics processor when dealing with the complexity of the current scene.

[0078] S403. The timing comparison module extracts a pre-set fixed millisecond value at the physical level as a tolerance threshold, and extracts a pre-set interpolable compensation threshold. The calculated absolute time difference is then used as the basis for the comparison. The results are compared and verified against the tolerance threshold and the interpolable compensation threshold to generate a time-series verification result.

[0079] To prevent the rendering scheduling mechanism from overreacting to minor time fluctuations, a tiered fault tolerance standard was introduced.

[0080] In this embodiment, the tolerance threshold is set based on the physiological tolerance limit of human visual and auditory perception to audio-visual misalignment. For example, the fixed millisecond value of the tolerance threshold can be set to 20 milliseconds to 40 milliseconds.

[0081] In addition, the interpolation compensation threshold is set based on the mathematical smoothness limit of the trajectory interpolation algorithm and the maximum acceptable level of human eye to screen jumps. For example, the interpolation compensation threshold can be set to 100 milliseconds to 150 milliseconds.

[0082] Based on the aforementioned judgment criteria, the logic checker built into the timing comparison module verifies the absolute time difference. Execute multi-level comparison logic:

[0083] When determining the absolute time difference When the value is less than or equal to the tolerance threshold, it indicates that the deviation between the current rendering progress of the image and the audio master data is within the safe zone of human perception, and a timing verification result indicating normal rendering is generated.

[0084] When determining the absolute time difference When the value is greater than the tolerance threshold and less than or equal to the interpolation compensation threshold, it indicates that there is an objective delay in the parsing of the visual discrete data, but the amount of delay is still within the range that subsequent algorithms can compensate for. Based on this, a time-series verification result indicating the need for interpolation compensation is generated.

[0085] When determining the absolute time difference When the time deviation exceeds the interpolation compensation threshold, it indicates that the device is experiencing a computing bottleneck or data blockage. Conventional interpolation calculations can no longer smoothly catch up with this time deviation, thus generating a timing verification result indicating the need for dimensionality reduction rendering. The timing verification results of the above different states will be transmitted to the next process in real time as decision instructions for reconstructing the final rendered screen.

[0086] In this embodiment, step S5 involves the playback scheduling module determining the final rendering method for the visual discrete data based on the aforementioned temporal comparison results. When rendering delays occur, this step aims to reduce temporal lag by reconstructing spatial coordinates or simplifying calculations. Step S5 specifically includes:

[0087] S501, the playback scheduling module receives the timing verification result generated in step S403; when the timing verification result indicates normal rendering, the visual discrete data is pushed normally for regular rendering according to the differentiated hardware rendering pipeline constructed in step S102, and normal synchronous playback is completed.

[0088] Normal rendering means that the current hardware computing power is sufficient, and the original graphics application programming interface call logic is maintained.

[0089] As a preferred underlying implementation, the vertex coordinates and texture mapping data of the visual discrete data are sequentially sent to the video memory buffer, and the graphics processor performs standard rasterization and pixel shading operations according to a preset frame refresh rate. The visual discrete data is presented in the order of its assigned linear timestamps, thus maintaining synchronization with the audio stream.

[0090] S502, When the playback scheduling module indicates that interpolation compensation is needed based on the timing verification result, it discards the lagging historical coordinate points for the cursor trajectory and handwriting in the visual discrete data, and calls the trajectory compensation formula to directly map the coordinate state that the current time node should have in the spatial dimension, so that the coordinate state is consistent with the audio timestamp obtained in step S401. Align and generate target space coordinate points and the target spatial coordinates point Push it to the rendering queue to complete the image compensation rendering.

[0091] Because the rendering progress of the screen lags behind the audio main axis, continuing to render the lagging coordinate points accumulated in the buffer queue could easily cause a visual misalignment between the content displayed on the screen and the sound. Therefore, coordinate data with timing attributes earlier than the current audio timestamp is directly removed from the rendering queue.

[0092] To address the screen abruptness issue caused by coordinate culling, the nearest successfully rendered historical reference coordinate point and the next target reference coordinate point that is later in time are extracted. The trajectory compensation formula established using a second-order Bézier curve algorithm is then used to calculate the spatial coordinates required for the current moment. Furthermore, if the current trajectory is in its initial stage and there is no successfully rendered historical reference coordinate point yet, the starting point coordinates of the trajectory are directly used as the target spatial coordinate point to avoid the algorithm getting stuck in a dead zone due to a lack of boundary conditions. The trajectory compensation formula is:

[0093] ;

[0094] in, ;

[0095] In the formula: Pointer to the current audio timestamp The generated target spatial coordinate points; This is the current audio timestamp; The time normalization coefficient; This is the historical reference coordinate point that was previously successfully rendered; Based on and The Bessel control points are derived from the motion vector calculations. The target reference coordinate point is located at a later time sequence. Historical baseline coordinates The corresponding absolute timestamp; Target reference coordinates The corresponding absolute timestamp.

[0096] Regarding the Bézier control points in the above formula The method of obtaining the data is derived from the motion vector formed by the instantaneous velocity and acceleration of the historical reference coordinate point at the time of recording.

[0097] Specifically, extraction At least one reference coordinate point before the point The displacement vector per unit time is calculated as the instantaneous velocity vector. The calculation formula is as follows:

[0098] ;

[0099] In the formula: Reference coordinate point The corresponding absolute timestamp.

[0100] Subsequently, along this instantaneous velocity vector The direction is according to the preset proportional coefficient Extend the reach to establish control points. The calculation formula is as follows:

[0101] ;

[0102] In this embodiment, a preset scaling factor is used. The value ranges from 0.3 to 0.5. This dynamic derivation mechanism allows the generated interpolation curve to better match the user's actual writing or sliding inertia. The target spatial coordinates are calculated... Then, it is directly pushed into the rendering pipeline to complete the drawing of the current frame.

[0103] S503 When the timing verification result indicates that dimensionality reduction rendering is required, the playback scheduling module directly abandons the calculation logic of the trajectory compensation formula for the cursor trajectory and handwriting in the visual discrete data to perform dimensionality reduction rendering. It only pushes the core node coordinates of the cursor trajectory and handwriting discrete to the rendering queue to complete the dimensionality reduction synchronous playback, so as to maintain the synchronization of the timing node with the master clock source established in step S401.

[0104] When the time delay exceeds the correctable range of interpolation compensation, it indicates that the current hardware load is already at its limit. At this time, performing floating-point interpolation operations will further consume processor resources and exacerbate the stuttering phenomenon.

[0105] To address this situation, a dimensionality reduction rendering strategy is implemented, stopping the calculation of all intermediate transition points on the smooth curve. In this embodiment, the original trajectory data to be rendered is traversed, and only the stroke start point, direction change point, and stroke end point are extracted as the core node coordinates.

[0106] The aforementioned directional change point refers to the inflection point where the angle between two adjacent trajectory segments changes more than a preset angle threshold. This preset angle threshold can be calibrated according to the actual writing characteristics, for example, set to 30 degrees to 45 degrees.

[0107] After extraction, the coordinates of these discrete core nodes are directly fed into the graphics processor's rendering pipeline and quickly displayed on the screen as straight-line connections. This operation reduces the amount of vertex processing required by the graphics processor, and in a computing-constrained environment, it maintains the synchronization between the screen progress and the master clock source by sacrificing the visual smoothness of the trajectory.

[0108] See appendix Figure 3 This invention also provides a time-indexed multimedia data synchronous playback device, including a data parsing module, a playback scheduling module, a cache control module, an audio / video rendering module, and a timing comparison module. The modules communicate with each other via an internal bus or data interface to collaboratively complete the multimedia data processing flow from low-level parsing, pipeline reconstruction, predictive scheduling to final synchronous rendering.

[0109] The data parsing module receives discretized multimedia files from external input during the initialization phase, extracts the unique logical identifier of the discretized multimedia file to determine its application scenario characteristics. Simultaneously, the module separates the discretized multimedia file into main audio data and visual discrete data, including static images, cursor trajectories, handwriting, micro-videos, and external materials, and extracts the mixed timestamp information bound to each type of visual discrete data.

[0110] In practice, the data parsing module extracts the hour, minute, and second portions of the mixed timestamp information, multiplies them by the corresponding millisecond conversion rates, and calculates the base millisecond value, which is unaffected by the video frame rate. It then reads the frame number offset (less than one second) or the absolute presentation time offset mapped from the mixed timestamp information to the underlying audio sampling sequence. Subsequently, the data parsing module adds the absolute presentation time offset to the base millisecond value to generate a linear timestamp, and assigns this linear timestamp to the corresponding visual discrete data as a temporal attribute.

[0111] The playback scheduling module is used to dynamically reconstruct the hardware resource allocation strategy at the underlying level based on the application scenario feature information determined by the data parsing module. It constructs a differentiated hardware rendering pipeline that matches the application scenario feature information by adjusting the concurrent processing priority of the graphics processor.

[0112] In practice, the playback scheduling module matches the extracted unique logical identifier with the preset scene mapping rules:

[0113] If the exclusive logical identifier matches a high-frequency dynamic interaction type, it confirms that the application scenario feature information represents a high-frequency video interaction scenario, thereby increasing the allocation weight of the hardware video decoder thread and prioritizing the memory throughput of the micro-video rendering unit.

[0114] If the exclusive logical identifier matches the static display type, it is confirmed that the application scenario feature information represents a low-frequency static whiteboard scenario, and some video decoding cores are actively suspended. The released computing resources are then concentrated on the anti-aliasing and low-latency rendering channels used to process handwritten annotation sequences.

[0115] The cache control module is used for backward sliding prediction of the time axis based on the linear timestamp. It uses the load density formula to calculate the multimedia data rendering pressure within the future pre-read time window and obtains the predicted load density.

[0116] When the predicted load density is less than or equal to a pre-set safety threshold, the module confirms that the predicted load density is a low load distribution and triggers a memory reclamation instruction to actively reclaim idle image memory. When the predicted load density is greater than the pre-set safety threshold, the module confirms that the predicted load density is a high load distribution and performs spatial dimension scheduling accordingly. It prioritizes the establishment of a high-priority buffer channel in memory for the upcoming visual discrete data and loads the rendering shader corresponding to the visual discrete data into the graphics processor cache in advance to complete the warm-up.

[0117] The audio and video rendering module includes an audio processing unit. This unit acquires the separated main audio data and enables it to play continuously at its native sampling rate.

[0118] The audio / video rendering module establishes the playback clock of the audio master data as the master clock source, which serves as the reference for the synchronization and alignment of the visual discrete data. Simultaneously, the audio processing unit outputs the current audio timestamp to the timing comparison module in real time. Furthermore, the audio / video rendering module also receives the visual discrete data and coordinate nodes pushed by the playback scheduling module to perform the final synchronized rendering of the visuals.

[0119] The timing comparison module receives the audio timestamp output by the audio processing unit and the linear timestamp assigned to the visual discrete data. It uses the linear timestamp as the target start and end timestamps and calculates the absolute time difference between the audio timestamp and the target start and end timestamps.

[0120] Furthermore, the timing comparison module extracts a pre-set fixed millisecond value as a tolerance threshold and a pre-set interpolation compensation threshold. The calculated absolute time difference is compared and verified with the tolerance threshold and the interpolation compensation threshold, thereby generating a timing verification result and outputting it to the playback scheduling module.

[0121] The playback scheduling module is also used to receive the timing verification results output by the timing comparison module, so as to perform dynamic rendering scheduling accordingly.

[0122] When the timing verification result indicates normal rendering, the playback scheduling module pushes the visual discrete data to the audio and video rendering module for regular rendering according to the differentiated hardware rendering pipeline.

[0123] When the timing verification result indicates that interpolation compensation is required, the module discards the lagging historical coordinate points for the cursor trajectory and handwriting in the visual discrete data, calls the trajectory compensation formula to directly map the coordinate state that the current time node should have in the spatial dimension, prompts the coordinate state to align with the audio timestamp to generate the target spatial coordinate point, and pushes the target spatial coordinate point to the rendering queue to complete the image compensation rendering.

[0124] When the timing verification result indicates that dimensionality reduction rendering is required, the module directly abandons the calculation logic of the trajectory compensation formula for the cursor trajectory and handwriting in the visual discrete data to perform dimensionality reduction rendering, and only pushes the discrete core node coordinates to the rendering queue of the audio and video rendering module, thereby completing the dimensionality reduction and synchronous playback.

[0125] To further verify the effectiveness of the present invention in practical application scenarios, this embodiment uses a typical online interactive remote experimental classroom as the application background.

[0126] In this scenario, the teacher uses an electronic whiteboard to handwrite derivations while simultaneously displaying a dynamic physics experiment simulation (micro-video) accompanied by live audio narration. Because the experiment simulation plugin consumes significant hardware resources, traditional playback methods often result in lag in the handwritten text.

[0127] In this embodiment, the discrete multimedia file is first parsed to identify a unique logical identifier, ".exp," representing a high-frequency interactive scenario. Based on this identifier, the allocation weight of the hardware video decoder thread is set to a higher level to ensure smooth decoding of the experimental micro-video.

[0128] A sudden load spike was detected after 1000 milliseconds of playback. At this point, the set pre-read time window duration... The timeframe is 2000 milliseconds (i.e., covering a period of 1000 to 3000 milliseconds). Within this window, 20 sampling points of handwritten handwriting data were detected for processing. The micro-video segment to be decoded is 1 channel ( The known computing power consumption weight coefficients for this scenario are as follows: and Substitute the data into the load density formula for calculation:

[0129] ;

[0130] The predicted load density is calculated. The value is 0.024. Given that the pre-set safety threshold is 0.015, 0.024 > 0.015, confirming a high load distribution. At this point, the cache control module immediately establishes a high-priority buffer channel in memory using direct memory access technology and preloads the experimental rendering shaders to be used into the graphics processor cache to warm them up and prepare for subsequent rendering pressure.

[0131] Subsequently, a timing comparison was performed. This was done at the audio timestamp. At milliseconds, the compensation implementation phase begins. At this point, the corresponding linear timestamp is obtained. Mapped to respectively milliseconds and Milliseconds. Absolute time difference due to deviations caused by load fluctuations. Milliseconds. The known tolerance threshold is 30 milliseconds, and the interpolation compensation threshold is 150 milliseconds. Since 30 < 40 ≤ 150, the interpolation compensation logic is triggered.

[0132] Extract historical reference coordinates and target reference coordinates The Bessel control points are calculated based on the motion vectors. First, calculate the time normalization coefficient:

[0133] ;

[0134] Substitute the above parameters into the trajectory compensation formula to calculate the coordinates that should appear at the current moment:

[0135] ;

[0136] ;

[0137] The calculated trajectory compensation point coordinates are (28.64, 26.24). This trajectory compensation point is directly pushed into the rendering queue. This forced alignment at the spatial level ensures a high degree of synchronization between the teacher's audio narration and the presentation of the handwritten ink.

[0138] See appendix Figure 4 , attached Figure 4 The mid-audio time axis reference is represented by a solid line and circular markers, showing an ideal linear growth; the traditional playback scheme is represented by a dotted line and square markers, which deviates significantly downward after a load burst occurs at 1000 milliseconds, reflecting that the rendering progress lags behind the sound; the synchronization scheme of this invention is represented by a dashed line and star markers.

[0139] As can be seen, the three lines overlapped before 1000 milliseconds. The synchronization scheme of this invention performs compensation at 1040 milliseconds. By calculating the generated trajectory compensation points (star-shaped markers), the rendering trajectory is forced to align and fit with the audio timeline reference within a short period of time. This proves that the scheme of this invention can solve the audio-visual misalignment problem through trajectory compensation under dynamic high-load environments, thereby improving the immediacy of the interactive experience.

[0140] To further verify the feasibility of the present invention and the rationality of the relevant formulas, this embodiment conducts a comparative experiment under a simulated high-load environment to quantitatively evaluate the synchronous playback effect.

[0141] This experiment aims to demonstrate that by removing video frame rate dependence through linear timestamps and by using load density formulas and trajectory compensation formulas for real-time intervention, the synchronization accuracy and smoothness of multimedia data during playback are significantly improved.

[0142] The specific steps of this experiment are as follows:

[0143] Prepare the experimental environment and materials. Select a computer with an integrated graphics processor as the test terminal and install the preset remote interactive teaching client. Select a 60-second discrete multimedia file with a unique logical identifier ".exp" as the experimental object. This file contains complex physics experiment micro-videos, high-density handwritten handwriting, and continuous audio master data.

[0144] Perform load stress testing. During the 1000-3000 millisecond time interval of multimedia file playback, start parallel graphics rendering tasks in the background of the terminal to artificially create computing power competition for the graphics processor, so as to simulate the high load distribution state in real application scenarios.

[0145] Collect and compare data. The background records the audio timestamps output by the audio processing unit in real time and simultaneously extracts the actual time node when the visual discrete data is pushed to the rendering queue. In traditional synchronization schemes, the presentation time of visual data is calculated from a fixed video frame rate; in the synchronization scheme of this invention, alignment is performed based on linear timestamps.

[0146] The formula calculation was verified. At the moment of the 1000-millisecond load burst, the predicted load density output by the load density formula reached 0.024, triggering the high-load cache intervention logic. At the subsequent 1040-millisecond node, since the absolute time difference reached 40 milliseconds, exceeding the 30-millisecond tolerance threshold, a trajectory compensation point was generated by calculating the trajectory compensation formula, achieving forced alignment of spatial coordinates.

[0147] The table below records the comparison results of key indicators between the traditional synchronization scheme and the synchronization scheme of this invention in the high load range of 1000 milliseconds to 3000 milliseconds:

[0148] Table 1. Experimental comparison of synchronization accuracy between traditional synchronization schemes and the synchronization scheme of this invention.

[0149] Traditional synchronization solutions 92 64.5 4.2 obvious Synchronization scheme of the present invention 38 26.2 0.8 weak

[0150] The experimental data shows that when faced with the same level of computing power bottleneck, the traditional synchronization scheme, due to its over-reliance on video frame rate and lack of forward-looking cache scheduling, has an absolute time difference peak of 92 milliseconds, which exceeds the tolerance limit of human perception, and on average, more than 4 rendering pauses occur per second.

[0151] In contrast, the synchronization scheme of this invention, through advance prediction of the load density formula, completes shader warm-up and the establishment of high-priority buffer channels before the arrival of high load. When timing deviations occur, the trajectory compensation formula, through spatial coordinate reconstruction, successfully suppresses the peak absolute time difference to 38 milliseconds, close to the tolerance threshold.

[0152] Experimental results demonstrate that the linear timestamp conversion logic and trajectory compensation formula in this invention are highly reasonable and can effectively maintain a high degree of consistency between visual discrete data and audio master data without increasing additional hardware costs.

Claims

1. A method for synchronously playing multimedia data based on time index, characterized in that, Includes the following steps: Extract the unique logical identifier of the discretized multimedia file to determine the application scenario feature information, adjust the concurrent processing priority of the graphics processor according to the application scenario feature information, and construct a differentiated hardware rendering pipeline. Separate the discretized multimedia file to obtain audio master data and visual discrete data, calculate the mixed timestamp information bound to the visual discrete data to generate a linear timestamp, and assign it as the temporal attribute of the visual discrete data; Based on the linear timestamp, the predicted load density is calculated using the load density formula, and low-load cache intervention or high-load cache intervention is performed according to the relationship between the predicted load density and the security threshold. Run the audio master data to establish the master clock source and obtain the audio timestamp, calculate the absolute time difference between the audio timestamp and the linear timestamp which serves as the target start and end timestamps, and compare and verify it with the tolerance threshold and the interpolable compensation threshold to generate the timing verification result; Based on the timing verification results: When normal rendering is indicated, regular rendering is performed according to the differentiated hardware rendering pipeline. When interpolation compensation is required, the trajectory compensation formula is invoked to map the target space coordinates. When a dimensionality reduction rendering is required, the coordinates of the core nodes are pushed to perform the dimensionality reduction rendering, thus completing the dimensionality reduction and synchronous playback.

2. The multimedia data synchronous playback method based on time indexing according to claim 1, characterized in that, The step of adjusting the concurrent processing priority of the graphics processor based on the application scenario feature information and constructing a differentiated hardware rendering pipeline includes: The extracted exclusive logical identifier is matched with the preset scene mapping rules; If the match is a high-frequency dynamic interaction type, it is confirmed that the application scenario feature information represents a high-frequency video interaction scenario, thereby increasing the thread allocation weight of the hardware video decoder and ensuring the video memory throughput of the micro-video rendering unit. If the match is a static display type, it is confirmed that the application scenario feature information represents a low-frequency static whiteboard scenario. Part of the video decoding core is suspended, and the released computing resources are allocated to the anti-aliasing and low-latency rendering channels for processing handwritten annotation sequences. The differentiated hardware rendering pipeline is constructed based on the different application scenario characteristics.

3. The multimedia data synchronous playback method based on time indexing according to claim 1, characterized in that, The step of calculating the hybrid timestamp information bound to the visual discrete data to generate a linear timestamp includes: Multiply the hour, minute, and second portions of the mixed timestamp information by the millisecond conversion rate to calculate the base millisecond value; Read the frame number offset of less than one second or the absolute presentation time offset of the audio sampling sequence from the mixed timestamp information; The linear timestamp is generated by adding the absolute presentation time offset to the base millisecond value.

4. The multimedia data synchronous playback method based on time indexing according to claim 1, characterized in that, The step of calculating the predicted load density based on the linear timestamp using the load density formula includes: Based on the time axis established by the linear timestamp, the number of occurrences of each of the visual discrete data or the volume of data to be processed within the pre-reading time window is predicted. Retrieve the computational power consumption weighting coefficients assigned to various types of visual discrete data based on the differentiated hardware rendering pipeline; The predicted load density is obtained by multiplying the various computing power consumption weight coefficients by the corresponding occurrence quantity or the volume of data to be processed, dividing each product by the pre-reading time window duration, and then summing them up. The duration of the pre-read time window is set based on the storage bus read rate and the average transmission bit rate of the discrete multimedia file.

5. The multimedia data synchronous playback method based on time indexing according to claim 1, characterized in that, The steps of performing low-load caching intervention or high-load caching intervention based on the relationship between the predicted load density and the security threshold include: When the predicted load density is less than or equal to the safety threshold, it is confirmed as a low load distribution, and a video memory reclamation instruction is triggered to reclaim idle image video memory, thus completing the low load cache intervention. When the predicted load density is greater than the safety threshold, it is confirmed as a high load distribution. A high-priority buffer channel for the visual discrete data is established in memory through spatial dimension scheduling, and the rendering shader is preloaded into the graphics processor cache to complete the high load cache intervention. The security threshold is set based on the maximum concurrent computing power of the graphics processor and the available physical memory.

6. The multimedia data synchronous playback method based on time indexing according to claim 1, characterized in that, The step of comparing and verifying the results with the tolerance threshold and the interpolable compensation threshold to generate the time series verification results includes: When the absolute time difference is less than or equal to the tolerance threshold, the timing verification result indicating normal rendering is generated; When the absolute time difference is greater than the tolerance threshold and less than or equal to the interpolable compensation threshold, the timing verification result indicating that interpolation compensation is required is generated; When the absolute time difference is greater than the interpolable compensation threshold, the timing verification result indicating that dimensionality reduction rendering is required is generated. The tolerance threshold is set based on the physiological tolerance for audio-visual misalignment, and the interpolable compensation threshold is set based on the mathematical smoothing limit of the trajectory interpolation algorithm and the acceptability of screen jumps.

7. The multimedia data synchronous playback method based on time indexing according to claim 1, characterized in that, When the indication requires interpolation compensation, the steps of calling the trajectory compensation formula to map the target space coordinate points include: For the cursor trajectory and handwriting in the visual discrete data, discard the lagging historical coordinate points; Using the historical reference coordinates, target reference coordinates, Bezier control points, and time normalization coefficients that have been successfully rendered, the current node coordinate state is mapped in the spatial dimension through the trajectory compensation formula. Align the coordinate state with the audio timestamp, generate the target spatial coordinate point, and push it to the rendering queue.

8. The multimedia data synchronous playback method based on time indexing according to claim 7, characterized in that, The steps for performing dimensionality reduction rendering on the core node coordinates of the push notification to achieve synchronized playback include: For the cursor trajectory and handwriting in the visual discrete data, the calculation of the trajectory compensation formula is abandoned in order to perform the dimensionality reduction rendering; Traverse the original trajectory data to be rendered, and extract the stroke start point, direction change point and stroke end point to form the core node coordinates; The discrete coordinates of the core nodes are pushed to the rendering queue, so that the rendering progress of the visual discrete data is synchronized with the master clock source.

9. The multimedia data synchronous playback method based on time indexing according to claim 8, characterized in that, The step of extracting the stroke start point, direction change point, and stroke end point to form the coordinates of the core node includes: The original trajectory data obtained through traversal is divided into several connected trajectory segments; Calculate the change in the angle between two adjacent trajectory segments in the plurality of connected trajectory segments; When the change in the included angle is greater than a preset angle threshold, the intersection point between the two adjacent trajectory segments is established as the direction change point. Extract the starting point of the stroke, the established direction change point, and the ending point of the stroke, and combine the coordinates of each point to form the coordinates of the core node; The preset angle threshold is pre-set based on the actual writing characteristics.

10. A multimedia data synchronization playback device based on time indexing, characterized in that, The multimedia data synchronization playback method based on time indexing, applied to any one of claims 1-9, includes: The data parsing module is used to extract unique logical identifiers to determine application scenario feature information, separate discretized multimedia files to obtain audio master data and visual discrete data, and calculate mixed timestamp information to generate linear timestamps assigned to the visual discrete data. A cache control module is used to calculate the predicted load density based on the linear timestamp using the load density formula, and to perform corresponding low-load cache intervention or high-load cache intervention. The audio and video rendering module contains an audio processing unit, which is used to run the audio master data to establish the master clock source and obtain the audio timestamp, and to receive the pushed data to perform screen rendering. A timing comparison module is used to calculate the absolute time difference between the audio timestamp and the linear timestamp, which serves as the target start and end timestamps, and to generate timing verification results by comparing a tolerance threshold and an interpolable compensation threshold. The playback scheduling module is used to construct a differentiated hardware rendering pipeline based on the application scenario feature information, and to perform regular rendering, compensated rendering, or push core node coordinates to perform dimensionality reduction rendering based on the timing verification results.

Images