Music on-demand system based on interactive virtual scene

Abstract

The present application relates to the technical fields of multimedia interaction and digital image processing, and particularly relates to a music on-demand system based on an interactive virtual scene.The present application obtains the expected output time of a visual picture in combination with the time consumption of a task backlog and the expected time consumption of asset hard disk loading for any instruction freezing duration.A one-dimensional Gaussian probability density function is constructed and integrated to obtain the cumulative probability distribution of visual output time according to the expected output time of the visual picture and the graphics processor lag index.A step cumulative probability distribution is obtained according to the accompaniment retake time point sequence and the expected output time of the visual picture.The sound-picture synchronization deviation value is calculated through the area difference between the two probability distributions.The sound-picture synchronization deviation value is attenuated by the graphics processor lag index, and the global comprehensive value is obtained in combination with the instruction freezing duration, and the optimal instruction freezing duration is selected from all instruction freezing durations according to the global comprehensive value, and the user instruction is executed in combination with the system timer.

Description

Technical Field

[0001] This invention relates to the field of multimedia interaction and digital image processing technology, specifically to a music on-demand system based on an interactive virtual scene. Background Technology

[0002] In interactive virtual scene-based music streaming systems (such as holographic KTV booths or immersive music games), users typically trigger 3D scene switching commands via touchscreen terminals. Upon receiving the command, the system loads the corresponding 3D model assets and presents holographic effects on the projection medium. Because the auditory channel in such scenes must support real-time microphone monitoring of the user's voice, the system cannot arbitrarily pause or delay the output of the accompaniment audio while waiting for the screen to load. Therefore, when a user taps the screen at random times, the fluctuating screen display timing often deviates from the accent points of the accompaniment audio, disrupting the rhythmic consistency of the live audiovisual experience.

[0003] Current technologies for audio-visual alignment in music streaming systems typically calculate the absolute time difference between the estimated video rendering time and the target audio beat time. However, in actual streaming terminals, the graphics processor's rendering pipeline experiences parallel task backlogs, and the process of reading 3D assets from the hard drive is easily affected by operating system scheduling, resulting in a jitter of tens of milliseconds in the final video rendering time. If the aforementioned point-to-point time difference comparison method is used, this slight hardware jitter can cause the system's alignment target to irrationally and frequently jump between two adjacent musical beats. Furthermore, if the system continues to blindly suspend user touch commands for extended periods while the device is under high load and experiencing lag, it will severely disrupt the basic interactive experience during scene transitions and may even cause users to perceive the interface as frozen. Summary of the Invention

[0004] To address the technical problems of poor audio-visual synchronization and long user command response times caused by fluctuations in hardware rendering time when handling audio-visual alignment in existing music playback systems, the present invention aims to provide a music playback system based on an interactive virtual scene. The specific technical solution adopted is as follows: This invention proposes a music on-demand system based on an interactive virtual scene, the system comprising: The data acquisition module is used to obtain the corresponding accompaniment rebeat time sequence, task backlog time, graphics processor lag index, and estimated time for loading assets from the hard drive based on the interaction trigger timestamps of user-generated commands; it also presets multiple command freeze durations. The objective function acquisition module is used to obtain the expected output time of the visual image by combining the task backlog time and the estimated time of loading the asset hard disk for any instruction freeze duration; construct a one-dimensional Gaussian probability density function and perform integration operation based on the expected output time of the visual image and the graphics processor stuttering index to obtain the cumulative probability distribution of the visual output time; and obtain the step cumulative probability distribution based on the accompaniment re-beat time sequence and the expected output time of the visual image. The global cost generation module is used to obtain the audio-visual synchronization deviation cost based on the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step; to attenuate the audio-visual synchronization deviation cost based on the graphics processor stuttering index; and to obtain the global comprehensive cost based on the instruction freeze duration. The instruction execution module is used to select the optimal instruction freeze duration from all instruction freeze durations based on the global comprehensive cost value obtained for each instruction freeze duration, and then execute user instructions in conjunction with the system timer.

[0005] Furthermore, the graphics processor stuttering index includes: Based on the interaction trigger timestamp, extract the actual time consumption record of the holographic projection image of the most recently continuously output preset number of frames in the underlying state snapshot of the system at that moment; The time variance in the actual time consumption record is used as the graphics processor stuttering index.

[0006] Furthermore, the preset multiple instruction freeze durations include: Initialize an instruction freeze duration for traversal testing, a step count index for loop calculation, and a fixed step size for traversal advancement in system memory; within a preset time interval, continuously increment the instruction freeze duration according to the fixed step size to obtain multiple instruction freeze durations.

[0007] Furthermore, the step of obtaining the expected output time of the visual image by combining the freeze duration of any instruction with the backlog time of the task and the estimated time of loading the asset hard drive includes: The interaction trigger timestamp is added to the task backlog time and the current instruction freeze time, respectively. The maximum sum of the two sums is then added to the estimated time for loading the asset hard drive to obtain the expected output time of the visual image.

[0008] Further, obtaining the step cumulative probability distribution based on the accompaniment re-beat time sequence and the expected output time of the visual image includes: Calculate the absolute time difference between each stressed beat time point in the accompaniment stressed beat time point sequence and the expected output time of the visual image; extract the stressed beat time point with the smallest absolute time difference as the target accompaniment stressed beat time point. Based on the target accompaniment downbeat time point, a step cumulative probability distribution is constructed in the form of an indicator function; before the target accompaniment downbeat time point, the step cumulative probability distribution outputs 0; after the target accompaniment downbeat time point, the step cumulative probability distribution outputs 1.

[0009] Further, obtaining the audio-visual synchronization deviation cost based on the area difference between the visual output time cumulative probability distribution and the step cumulative probability distribution includes: Using the interaction trigger timestamp as the lower limit of integration and the preset dynamic safety timestamp as the upper limit of integration, definite integrals are performed on the cumulative probability distribution of visual output time and the cumulative probability distribution of step, to obtain the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step on the interval formed by the lower limit of integration and the upper limit of integration, which is used as the cost of audio-visual synchronization deviation.

[0010] Furthermore, the attenuation of the audio-visual synchronization deviation cost based on the graphics processor stuttering index includes: The rendering jitter attenuation coefficient is calculated by multiplying the graphics processor stuttering index by a preset attenuation constant and using the power of an exponential function with the natural constant as the base. The attenuation deviation value is obtained by multiplying the rendering jitter attenuation coefficient by the audio-visual synchronization deviation value.

[0011] Furthermore, obtaining the overall comprehensive cost value includes: The difference between the current command freeze duration and the preset visually imperceptible waiting time limit is input into the one-sided penalty calculation model with non-negative boundary constraints, and the output result is used as the interaction delay cost of the current command freeze duration. The attenuation deviation cost and the interaction delay cost are weighted and summed based on preset delay dimension balance coefficients and area dimension balance coefficients to obtain the global comprehensive cost.

[0012] Furthermore, the step of selecting the optimal instruction freeze duration from all instruction freeze durations based on the global comprehensive cost value obtained for each instruction freeze duration includes: Using a conventional array traversal comparison algorithm, the global comprehensive value with the smallest value is selected in chronological order, and the corresponding instruction freeze duration is taken as the optimal instruction freeze duration.

[0013] Furthermore, the execution of user instructions in conjunction with the system timer includes: The optimal instruction freeze duration is used as the countdown time parameter for the system timer, and a countdown operation is performed. Maintain a continuous interruption counter in system memory, initialized to zero, and set a threshold for the number of interruptions; If the user does not generate a new command during the countdown, the user's current command will be executed when the countdown ends; if the user generates a new command, the system will immediately trigger the continuous interruption counter, and each new command will increment the number of consecutive interruptions by 1. If the cumulative number of consecutive interruptions reaches the threshold, the system determines that the user is in a rushing state, forcibly stops the current countdown, and immediately executes the user's current instruction. If the cumulative number of consecutive interruptions is less than the threshold, the system determines that the user is making a normal intention change, destroys the user's current instruction, recalculates the optimal instruction freeze time based on the latest user instruction, and executes the latest user instruction.

[0014] The present invention has the following beneficial effects: This invention, for any instruction freeze duration, combines the task backlog time and the estimated time for loading the asset's hard drive to obtain the expected output time of the visual image, estimating the most likely moment when the target image is output. Based on the expected visual image output time and the graphics processor stuttering index, a one-dimensional Gaussian probability density function is constructed to quantify the uncertainty of the final image presentation time. Integrating the one-dimensional Gaussian probability density function yields the cumulative probability distribution of the visual output time, representing the probability interval of the actual image presentation time. Based on the accompaniment re-beat time sequence and the expected visual image output time, a step cumulative probability distribution is obtained, representing the ideal moment for audio-visual synchronization.

[0015] Based on the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step, the audio-visual synchronization deviation cost is obtained, quantifying the degree of misalignment between the actual time probability interval of the image presentation and the ideal presentation moment of audio-visual synchronization. The audio-visual synchronization deviation cost is attenuated based on the graphics processor stuttering index to prevent excessively long user command response times when the system background is under high load and congestion. Combined with the command freeze duration, a global comprehensive cost is obtained, comprehensively evaluating and balancing the compromise point between the conflicting user experience dimensions of user command response speed and audio-visual synchronization accuracy, thus quantifying the overall user sensory experience.

[0016] Among all instruction freeze durations, the optimal instruction freeze duration with the lowest global comprehensive cost value is selected based on the global comprehensive cost value obtained for each instruction freeze duration. This ensures that the system balances user instruction response speed and audio-visual synchronization accuracy while guaranteeing the best overall user experience. User instructions are then executed in conjunction with the system's timer's delayed emission mechanism. Attached Figure Description

[0017] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural diagram of a music on-demand system based on an interactive virtual scene, provided as an embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a music on-demand system based on an interactive virtual scene proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The following description, in conjunction with the accompanying drawings, details a specific solution for a music on-demand system based on an interactive virtual scene provided by the present invention.

[0022] Please see Figure 1 The diagram illustrates a structure of a music on-demand system based on an interactive virtual scene according to an embodiment of the present invention. The system includes: a data acquisition module 101, an objective function acquisition module 102, a global cost generation module 103, and an instruction execution module 104.

[0023] The data acquisition module 101 is used to obtain the corresponding accompaniment rebeat time sequence, task backlog time, graphics processor lag index and estimated time for loading asset hard disk based on the interaction trigger timestamp of the user's command action; and preset multiple command freeze durations.

[0024] To ensure audio-visual synchronization and user operation command response efficiency in interactive virtual music playback scenarios (such as holographic KTV stores or immersive music games), the core requirement is to combine the alignment between the actual rendering time of the music playback system and the audio repetition time, completing audio-visual synchronization and user operation command response within the user's visual tolerance time. Therefore, the steps in this embodiment provide a data foundation for subsequent analysis.

[0025] Interaction trigger timestamp refers to the absolute system time of the user's command action (such as clicking the song skip button on the music playback system screen to skip a song) recorded by the system with millisecond precision.

[0026] The sequence of accented beats in the accompaniment serves as a benchmark for subsequent calculations of audio-visual synchronization deviation. Each accented beat represents the ideal moment when the image should appear.

[0027] Task backlog time represents the base time that user command tasks must wait in a queue if user commands are issued immediately at the current moment.

[0028] The Graphics Processing Unit (GPU) stutter index quantifies the jitter in rendering time across the GPU's rendering pipeline, indicating the level of stuttering on the current device. A higher GPU stutter index indicates more stuttering, while a lower index indicates smoother performance. This can be quantified using data such as GPU utilization.

[0029] The estimated time for loading assets from the hard drive represents the estimated time required for the system to read the 3D model asset corresponding to the user's command target from the hard drive into the video memory when executing the user's command.

[0030] The command freeze duration is a candidate waiting time intentionally delayed by the system when issuing user commands, i.e., a delay strategy. In subsequent algorithm derivation, the system will select the optimal command freeze duration from multiple options while ensuring user experience.

[0031] In some implementations of this invention, since the styles of the tracks in the on-demand music library vary and the overall levels of the audio signals differ, using a fixed threshold to determine the beat could lead to missed beats in gentle songs or incorrect identification of ordinary drum beats in heavy metal songs. Therefore, this invention determines beats by calculating a dynamic beat identification threshold within a local time window and constructing a sequence of beat time points in the accompaniment based on the absolute system time of the beat.

[0032] Specifically, taking the interaction trigger timestamp of the user's current command action as the starting reference point, the system extracts future accompaniment audio data from the pre-decoded audio buffer pool with a preset fixed duration as a time-domain slice window. The range of the time-domain slice window is [interaction trigger timestamp, interaction trigger timestamp + preset fixed duration of the pre-decoded audio buffer pool].

[0033] The short-time Fourier transform (SFT) algorithm is used to perform time-frequency conversion on the future accompaniment audio data within a time-domain slice window. The resulting two-dimensional time-frequency matrix has the time frame as the horizontal axis and the frequency band as the vertical axis. Each element in the matrix is ​​obtained by taking the absolute value of the SFT algorithm result, representing the spectral amplitude or energy value of the corresponding frequency unit at a specific time frame. The low-frequency energy envelope curve of the 20Hz to 250Hz frequency band in the two-dimensional time-frequency matrix is ​​extracted. This frequency range typically includes the core instrument signals such as the kick drum and bass, which constitute the heavy beat of the music. It should be understood that the method of time-frequency conversion using the SFT algorithm is a well-known technique and will not be elaborated here.

[0034] The energy values ​​of each time frame within the low-frequency energy envelope curve are summed to obtain the numerator. The total number of time frames contained within the time-domain slice window is used as the denominator. The resulting value is the local energy average of the low-frequency energy envelope curve within the time-domain slice window. This local energy average is multiplied by a preset empirical gain coefficient to obtain the dynamic repeat detection threshold. The system traverses the low-frequency energy envelope curve in chronological order, filtering out the corresponding time frames whose energy values ​​exceed the dynamic repeat detection threshold. These time frames correspond to the absolute system time. All corresponding time frames whose energy values ​​exceed the dynamic repeat detection threshold are sorted in chronological order to obtain the accompaniment repeat time point sequence.

[0035] In one specific implementation of this invention, the preset fixed duration of the pre-decoding audio buffer pool is set to 5000ms, and the preset empirical gain coefficient is 1.5.

[0036] Furthermore, to prevent errors such as division by zero crashes in subsequent mathematical deductions due to the lack of identified accents during the song's intro or light music sections without percussion, the system introduces a beat anomaly detection mechanism. The system performs a numerical check on the total number of accents. If the total number is zero, the system determines that there are no accents and skips all subsequent waiting deduction logic, forcibly setting the optimal instruction freeze duration to 0ms, i.e., triggering screen loading without any delay. If the total number is greater than zero, the system retains the generated accompaniment accent time sequence and outputs it to subsequent modules for later use.

[0037] In this embodiment of the invention, the system reads the current underlying state snapshot at the instant the interaction trigger timestamp is generated. The underlying state snapshot is formed by the graphics rendering engine (software) continuously writing the estimated total milliseconds required for the backlog of received but unprocessed image instructions in the current graphics processor's rendering pipeline, along with the time taken for historical output images, into high-speed memory at fixed intervals (e.g., every 50ms). The estimated total milliseconds required for the backlog of received but unprocessed image instructions in the current graphics processor's rendering pipeline, as seen in the underlying state snapshot, is used as the task backlog time.

[0038] In this embodiment of the invention, the system parses the switching scenario of the user command and extracts the identifier of the target 3D model asset to be called in the command. The system compares the identifier with the asset mapping table pre-stored on the local disk to extract the estimated time required to read the target 3D model asset from the hard disk into the video memory. This estimated time is used as the estimated time for loading the asset onto the hard disk.

[0039] An asset mapping table is a statically or dynamically updated data table, typically stored in a local configuration file or database. The table uses the identifier (ID) of the 3D model asset as the primary key, and stores the corresponding file size of the asset under standard hardware conditions and the estimated time required to load it from disk to video memory. It is generated when the asset is imported into the system or refreshed periodically based on historical average loading times.

[0040] The objective function acquisition module 102 is used to obtain the expected output time of the visual image by combining the task backlog time and the expected time of loading the asset hard disk for any instruction freeze duration; construct a one-dimensional Gaussian probability density function and perform integration operation based on the expected output time of the visual image and the graphics processor stuttering index to obtain the cumulative probability distribution of the visual output time; and obtain the step cumulative probability distribution based on the accompaniment rebeat time sequence and the expected output time of the visual image.

[0041] Because of rendering time jitter in the graphics processing unit's (GPU) rendering pipeline, the final presentation time of the image is not a fixed absolute point in time, but rather a time probability range. Traditional music playback systems, which use a fixed absolute point in time to determine audio-visual synchronization, may incorrectly delay the image that should appear on the current beat to the next beat, or advance the image that should match the next beat to the current beat, when normal fluctuations occur in rendering time. This causes the target visual elements (such as effects, text, and indicators) to jump back and forth between adjacent beats, severely disrupting the continuity and rhythm of audio-visual synchronization. The embodiments of this invention fundamentally solve this problem by constructing a cumulative probability distribution of visual output time and a cumulative probability distribution of step: the cumulative probability distribution of step represents the ideal presentation time of audio-visual synchronization, assuming that there is no rendering time jitter in the GPU's rendering pipeline, and the image is presented precisely and instantaneously at the target beat point. The cumulative probability distribution of visual output time takes into account the task queuing backlog in the current GPU rendering pipeline and the GPU stuttering index, resulting in the actual time probability range of the image presentation. Subsequent steps involve analyzing two probability distribution functions to smoothly absorb rendering time jitter in the graphics processor's rendering pipeline, eliminating visual jump defects caused by rendering time jitter, and achieving a stable and accurate audio-visual synchronization experience in a real hardware fluctuation environment.

[0042] The expected output time of the visual image represents the time when the system is most likely to output the target image after the user gives an instruction, after the system's delayed scheduling, the backlog of tasks in the graphics processor's rendering pipeline, and the loading of asset data.

[0043] Based on the expected visual output time and the graphics processing unit (GPU) stuttering index, a one-dimensional Gaussian probability density function is constructed to quantify the uncertainty of the final rendering time. The rendering time of the GPU's rendering pipeline is not a fixed value but fluctuates. If only the expected visual output time and the timing of the accompanying emote are compared, even if the rendering time fluctuates by only a few milliseconds, the system will consider it misaligned and continuously adjust the instruction freeze duration, causing the target visual element to jump back and forth between adjacent emotes. Constructing the Gaussian probability density function expands the expected visual output time from a single point of expectation into an interval. The larger the GPU stuttering index, the more severe the fluctuation in the rendering time of the GPU's rendering pipeline; the wider the corresponding interval, the higher the uncertainty of the final rendering time deviating from the expectation.

[0044] Since the final presentation of the image is a definite event on the timeline, and the ordinate of the one-dimensional Gaussian probability density function represents the rate of change of probability rather than the actual probability value, it cannot be directly compared numerically with a specific instant. Therefore, an integral operation along the timeline is performed on the one-dimensional Gaussian probability density function to obtain the cumulative probability distribution of the visual output over time. This distribution characterizes the temporal probability range of the actual presentation of the image. This operation avoids decision oscillations that might arise from using the probability density function for highly sensitive instantaneous value comparisons.

[0045] Based on the beat time of the target frame that is closest to the beat time of the accompaniment when executing the user command, a step cumulative probability distribution is obtained, which represents the ideal moment for audio-visual synchronization.

[0046] Specifically, the expected output time of the visual image is taken as the center of symmetry expectation, and the graphics processor stuttering index is taken as the variance. A one-dimensional Gaussian probability density function is constructed on a coordinate axis with time as the independent variable. The system performs an integral calculation on the one-dimensional Gaussian probability density function from left to right along the time axis, generating a probability function curve with a smooth S-shaped rise, which is the cumulative probability distribution of visual output time. The smoothness of this cumulative probability distribution of visual output time is positively correlated with the graphics processor stuttering index; the larger the graphics processor stuttering index (the more stuttering the device), the wider the climbing range of the S-shaped curve, that is, the wider the probability range of the actual time of image presentation.

[0047] The global cost generation module 103 is used to obtain the audio-visual synchronization deviation cost based on the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step; to attenuate the audio-visual synchronization deviation cost based on the graphics processor stuttering index; and to obtain the global comprehensive cost based on the instruction freeze duration.

[0048] The step cumulative probability distribution represents the ideal moment of audio-visual synchronization. It assumes that the graphics processor's rendering pipeline has no rendering time jitter, and the image is presented precisely and instantaneously at the target repetition point. The visual output time cumulative probability distribution considers the task queuing backlog in the current graphics processor's rendering pipeline and the graphics processor's stuttering index, resulting in the actual time probability range of image presentation. By analyzing the area difference between these two probability distribution functions, the degree of misalignment between the actual time probability range of image presentation and the ideal moment of audio-visual synchronization under the current instruction freeze duration is quantified, i.e., the cost of audio-visual synchronization deviation. The larger the cost of audio-visual synchronization deviation, the more severe the audio-visual synchronization misalignment will be due to the current latency strategy; the smaller the cost of audio-visual synchronization deviation, the more closely the current latency strategy will make the actual time probability range of image presentation closely match the ideal moment of audio-visual synchronization.

[0049] If the music streaming system's backend is under high load and congestion (e.g., downloading and caching a large number of tracks at high frequency), the graphics processor's rendering pipeline will inevitably experience significant rendering time jitter. At this point, the device will severely lag, and the actual display time will have a very wide probability range. The misalignment between the displayed image and the ideal timing of audio-visual synchronization will become very severe, resulting in a huge cost for audio-visual synchronization deviation. Without attenuation, the system, in order to eliminate the significant cost of this synchronization deviation as much as possible, will choose an extremely long command freeze duration, forcing the system to suspend commands for an extended period for a target that cannot be precisely timed. This causes user commands to remain unresponsive for a long time, creating the illusion of a system crash.

[0050] Therefore, the cost of audio-visual synchronization deviation is attenuated based on the graphics processor stuttering index. The larger the graphics processor stuttering index, the greater the attenuation of the audio-visual synchronization deviation cost; the smaller the graphics processor stuttering index, the smaller the attenuation of the audio-visual synchronization deviation cost. By attenuating the cost of audio-visual synchronization deviation, the system is forced to abandon the pursuit of precise audio-visual synchronization in the cost balance, and instead meet the efficiency of responding to user commands, thus achieving an adaptive deadlock prevention compromise.

[0051] The overall global cost-effectiveness (OBF) is a metric that, under the current command freeze duration, comprehensively evaluates and balances the conflicting user experience dimensions of command response speed and audio-visual synchronization accuracy. It quantifies the user's overall sensory experience. A higher BEF indicates a worse overall user experience. It is the sole criterion for subsequently identifying the optimal command freeze duration from multiple options while ensuring the best user experience.

[0052] The instruction execution module 104 is used to select the optimal instruction freeze duration from all instruction freeze durations based on the global comprehensive cost value obtained for each instruction freeze duration, and then execute the user instruction in conjunction with the system timer.

[0053] Since the system's task backlog time, estimated asset hard drive loading time, and graphics processor lag index are not fixed, a fixed latency strategy cannot adapt to all scenarios. Therefore, the system simulates the global comprehensive cost under different instruction freeze durations to select the latency strategy that best guarantees the overall user experience. Combined with the system timer's delayed emission mechanism, user instructions are registered as callback tasks to be executed after the system timer expires.

[0054] In summary, this invention obtains the corresponding accompaniment re-beat time sequence, task backlog time, graphics processor lag index, and estimated asset hard drive loading time by collecting the interaction trigger timestamps of user-generated commands. Multiple command freeze durations are also preset. For any command freeze duration, the expected visual output time is obtained by combining the task backlog time and the estimated asset hard drive loading time. Based on the expected visual output time and graphics processor lag index, a one-dimensional Gaussian probability density function is constructed and integrated to obtain the cumulative probability distribution of the visual output time. Finally, a step cumulative probability distribution is obtained based on the accompaniment re-beat time sequence and the expected visual output time.

[0055] The audio-visual synchronization deviation cost is obtained based on the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step. This cost is then attenuated using the graphics processor stuttering index, and combined with the instruction freeze duration to obtain the global comprehensive cost. The optimal instruction freeze duration is selected from all instruction freeze durations, and combined with the system timer's delayed emission mechanism, user instructions are executed.

[0056] Preferably, in some implementations of the embodiments of the present invention, the graphics processor stuttering index includes: The system extracts the actual time consumption records of the most recently continuously output holographic projection images of a preset number of frames from the underlying state snapshot at the time stamp of the user's interaction trigger. Standard statistical variance calculations are performed on these actual time consumption records, and the variance in these records is used as the graphics processor stuttering index. This index represents the degree of rendering time jitter in the graphics processor's rendering pipeline, i.e., the stuttering level of the current device. A higher graphics processor stuttering index indicates greater stuttering on the current device; a lower index indicates smoother performance.

[0057] In this embodiment of the invention, in order to avoid abnormal calculation of the graphics processor stuttering index due to insufficient accumulated frame count when the system is first started, if the number of frames output by the current graphics processor in the most recent consecutive period is less than the preset number of frames, the system directly uses the preset default anti-shake constant (e.g., 50ms²) as the current graphics processor stuttering index.

[0058] Furthermore, to prevent the actual time consumption records of the preset number of holographic projection images recently continuously output by the graphics processor from being completely identical when the device is extremely idle, resulting in zero time consumption variance in the actual time consumption records, zero graphics processor stuttering index, and thus causing the collapse of the subsequent steps in constructing the one-dimensional Gaussian probability density function, the system introduces an empirical variance threshold constraint mechanism of the underlying hardware.

[0059] Specifically, a preset empirical variance threshold is compared with the variance of time consumption in the current actual time consumption record. The system performs a maximum value operation, taking the maximum value between the preset empirical variance threshold and the variance of time consumption in the current actual time consumption record as the variance of time consumption in the current actual time consumption record.

[0060] In one specific implementation of this invention, the preset frame count is set to 100 frames, and the preset empirical variance threshold is set to 1ms².

[0061] Preferably, in some implementations of the embodiments of the present invention, multiple instruction freeze durations are preset, including: Initialize a variable in system memory for traversal testing, namely the instruction freeze duration. Within this variable system, set a loop-calculated step index, initially valued at 0. Based on the standard 60Hz refresh rate of an interactive screen, set a fixed step size for traversal progression to precisely match the natural frame interval of the display hardware. Within a preset time interval, continuously increment the instruction freeze duration by the fixed step size to obtain multiple instruction freeze durations. Each instruction freeze duration then proceeds to the subsequent module steps.

[0062] In one specific implementation of this invention, the fixed step size for traversal is set to 16.6ms, and the preset time interval is [0ms, 800ms].

[0063] Preferably, in some implementations of the present invention, for any instruction freeze duration, the expected output time of the visual image is obtained by combining the task backlog time and the estimated time for loading the asset hard drive, including: For any given command freeze duration, since the command freeze duration and the system task backlog time are concurrent phenomena that start simultaneously and overlap, the one that takes longer constitutes the final waiting bottleneck. Therefore, the maximum sum of the two sums (after adding the interaction trigger timestamp to each) is extracted and used as the bottleneck time baseline. Adding the bottleneck time baseline to the estimated asset disk loading time yields the expected visual output time corresponding to the current command freeze duration, representing the most likely time the visual will appear under the current command freeze duration.

[0064] Preferably, in some implementations of the present invention, obtaining the step cumulative probability distribution based on the accompaniment re-beat time sequence and the expected output time of the visual image includes: Calculate the absolute time difference between each stressed beat time point in the accompaniment stressed beat time point sequence and the expected output time of the visual image; extract the stressed beat time point with the smallest absolute time difference as the target accompaniment stressed beat time point. Based on the target accompaniment downbeat time point, a step cumulative probability distribution is constructed in the form of an indicator function; before the target accompaniment downbeat time point, the step cumulative probability distribution outputs 0; after the target accompaniment downbeat time point, the step cumulative probability distribution outputs 1.

[0065] The actual time the image appears should be attributed to the nearest accented beat in the accompaniment. Choosing the accented beat with the smallest absolute difference from the expected visual output time means the system can achieve audio-visual synchronization with minimal waiting time. Therefore, the step cumulative probability distribution constructed using the accented beat with the smallest absolute difference from the expected visual output time as the target accented beat is represented on the timeline as an ideal step with a vertical jump at the target accented beat, representing the ideal moment for audio-visual synchronization.

[0066] Preferably, in some implementations of the present invention, the audio-visual synchronization deviation cost is obtained based on the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step, including: Using the interaction trigger timestamp as the lower limit of integration and the preset dynamic safety timestamp as the upper limit of integration, definite integrals are performed on the cumulative probability distribution of visual output time and the cumulative probability distribution of step, to obtain the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step on the interval formed by the lower limit of integration and the upper limit of integration, which is used as the cost of audio-visual synchronization deviation.

[0067] To ensure that the cumulative probability distribution of visual output time can achieve normalization convergence of more than 99.7% (i.e., the function value approaches 1 infinitely) before the end of integration, thereby avoiding truncation errors in area calculation, a preset dynamic safety timestamp is introduced as the upper limit of integration.

[0068] In one specific implementation of this invention, the arithmetic square root of the graphics processor stuttering index is used as the standard deviation to calculate the values ​​of [expected visual output time + 3 times the standard deviation] and [interaction trigger timestamp + preset fixed duration of the pre-decoded audio buffer pool], and the maximum value of these is used as the preset dynamic security timestamp.

[0069] Specifically, the formula for calculating the audio-visual synchronization deviation cost using definite integrals is as follows: Where t represents any moment within the timestamp interval, and the timestamp interval is [ , ]; This represents the freeze duration of the k-th instruction; The value of the audio-visual synchronization deviation corresponding to the freeze duration of the kth instruction; The cumulative probability distribution of visual output time corresponding to the freeze duration of the kth instruction; This represents the cumulative probability distribution of the step jump corresponding to the freeze duration of the k-th instruction. Represents the timestamp of the interaction trigger; Represents a preset dynamic security timestamp; It represents the degree of misalignment between the actual time probability interval of the image presentation at any time t in the timestamp interval and the ideal time of audio-visual synchronization.

[0070] It should be understood that the above process of calculating area differences based on definite integrals is a well-known technique, and will only be briefly introduced here without further elaboration.

[0071] Preferably, in some implementations of the present invention, attenuating the audio-visual synchronization deviation cost based on the graphics processor stuttering index includes: The rendering jitter attenuation coefficient is calculated by using the product of the graphics processor stuttering index and the preset attenuation constant as the power of an exponential function with the natural constant as the base. Since the negative exponent operation maps the product of the graphics processor stuttering index and the preset attenuation constant to a rendering jitter attenuation coefficient between 0 and 1, the rendering jitter attenuation coefficient and the audio-visual synchronization deviation cost can be directly multiplied to obtain the attenuation deviation cost, thereby achieving attenuation.

[0072] The cost of audio-visual synchronization deviation is attenuated based on the graphics processor stuttering index, realizing the logic that the larger the graphics processor stuttering index, the greater the attenuation of the cost of audio-visual synchronization deviation; and the smaller the graphics processor stuttering index, the smaller the attenuation of the cost of audio-visual synchronization deviation.

[0073] Specifically, the formula for obtaining the rendering jitter attenuation coefficient is: in, This represents the rendering jitter attenuation factor; e represents the natural constant. Represents the preset attenuation constant; This represents the graphics processor's lag index.

[0074] The preset attenuation constant is used to adjust the system's sensitivity to changes in the graphics processor's stuttering index. In this embodiment, based on hardware environment benchmark testing experience, it is set to 0.005.

[0075] In negative exponents In this context, due to the graphics processor's lag index... Always non-negative, the product term must be a non-positive number. This negative exponent operation ensures that the generated rendering jitter attenuation coefficient is strictly mapped between (0, 1). The larger the graphics processor stuttering index, the closer the generated rendering jitter attenuation coefficient is to zero; the smaller the graphics processor stuttering index, the closer the generated rendering jitter attenuation coefficient is to 1.

[0076] Preferably, in some implementations of the embodiments of the present invention, obtaining the global comprehensive cost value includes: The difference between the current command freeze duration and the preset visually imperceptible waiting time limit is input into the one-sided penalty calculation model with non-negative boundary constraints, and the output result is used as the interaction delay cost of the current command freeze duration. The attenuation deviation cost and the interaction delay cost are weighted and summed based on preset delay dimension balance coefficients and area dimension balance coefficients to obtain the global comprehensive cost.

[0077] The difference between the current command freeze duration and the preset upper limit of visually imperceptible waiting time is used as the first deviation value of the one-sided penalty calculation model with non-negative boundary constraints. Since human vision has a tolerance of tens to hundreds of milliseconds for sluggishness during scene transitions, a one-sided truncation function is used to filter out the waiting time within the tolerance period, applying the penalty only to the portion exceeding the tolerance period. In the logic of the one-sided truncation function, if the first deviation value is less than or equal to zero (i.e., the command freeze duration does not exceed the preset upper limit of visually imperceptible waiting time), the function term forces the output to zero, indicating that the current command freeze duration is within the user's tolerance period; if the first deviation value is greater than zero, the function term outputs the first deviation value. The larger the first deviation value, the more the current command freeze duration exceeds the user's tolerance period, and the less tolerant the user is of it.

[0078] As can be seen from the steps described above for attenuating the cost of audio-visual synchronization deviation, the larger the graphics processor stuttering index, the closer the generated rendering jitter attenuation coefficient is to zero, and the closer the attenuation deviation cost obtained by attenuating the audio-visual synchronization deviation cost is to zero. At this time, the overall global cost is mainly determined by the interaction latency cost, and the user's overall sensory experience is determined by the user command response speed, with the system prioritizing the user command response speed. Conversely, the smaller the graphics processor stuttering index, the closer the generated rendering jitter attenuation coefficient is to 1, and the closer the attenuation deviation cost obtained by attenuating the audio-visual synchronization deviation cost is to the audio-visual synchronization deviation cost itself. At this time, the overall global cost is determined by both the interaction latency cost and the attenuation deviation cost, and the user's overall sensory experience is jointly determined by the user command response speed and the audio-visual synchronization accuracy, with the system simultaneously considering both aspects when responding to user commands. This calculation method ensures that the overall global cost can make a soft compromise between the user command response speed and the audio-visual synchronization accuracy based on real-time environmental fluctuations.

[0079] Since the cost of interaction delay is the product of the time deviation magnitude and the slope of the linear penalty (usually on a large order of magnitude), while the cost of attenuation deviation is obtained by attenuating the cost of audio-visual synchronization deviation, and the cost of audio-visual synchronization deviation is the area integral squeezed out by the probability distribution curve (usually on a small order of magnitude), to prevent the smaller value from being completely swallowed up by the larger value in terms of scale and losing its moderating effect during algebraic addition, a dimensional balance coefficient is introduced to unify the numerical scale.

[0080] Specifically, the one-sided penalty calculation model for obtaining the non-negative boundary constraint of the interaction delay cost is as follows: in, This represents the freeze duration of the k-th instruction; This represents the interaction delay cost under the freeze duration of the k-th instruction; This represents the preset linear penalty slope coefficient. The larger the value of this parameter, the more severe the system's penalty for excess latency. The first deviation value represents the one-sided penalty calculation model for non-negative boundary constraints; This represents the preset upper limit of visually imperceptible waiting time; max is the maximum value filtering function.

[0081] In one specific implementation of this invention, the preset linear penalty slope coefficient is set to 10, and the preset upper limit of visually imperceptible waiting time is set to 150ms.

[0082] It should be understood that the method of using a one-sided penalty calculation model with non-negative boundary constraints is a well-known technique, and will only be briefly introduced here without further elaboration.

[0083] Specifically, the formula for obtaining the global comprehensive cost value is: in, This represents the freeze duration of the k-th instruction; This represents the global total cost during the freeze duration of the k-th instruction. Represents the preset delay dimension balance coefficient; This represents the interaction delay cost under the freeze duration of the k-th instruction; The area dimension balance coefficient; The attenuation deviation value represents the attenuation deviation cost under the freeze duration of the k-th instruction, which is obtained from the audio-visual synchronization deviation value.

[0084] In one specific implementation of this invention, the delay dimension balance coefficient is set to 1, and the area dimension balance coefficient is set to 50.

[0085] Preferably, in some implementations of the present invention, the optimal instruction freeze duration is selected from all instruction freeze durations based on the global comprehensive cost value obtained for each instruction freeze duration, including: Within a preset time interval, the above-mentioned module calculates the freeze duration for each instruction, which increases in fixed steps, to obtain multiple instruction freeze durations and their corresponding global comprehensive value. Using a conventional array traversal comparison algorithm, the system selects the global comprehensive value with the smallest value in chronological order, and uses its corresponding instruction freeze duration as the optimal instruction freeze duration. At this point, the system balances user instruction response speed and audio-visual synchronization accuracy while ensuring the best overall user experience.

[0086] Preferably, in some implementations of the embodiments of the present invention, executing user instructions in conjunction with a system timer includes: Since the system's task backlog time, estimated asset hard drive loading time, and graphics processor lag index are not fixed, a fixed latency strategy cannot adapt to all scenarios. Therefore, the system simulates the global comprehensive cost under different instruction freeze durations to select the instruction freeze duration that best guarantees the overall user experience—the optimal instruction freeze duration. Combined with the system timer's delayed emission mechanism, the optimal instruction freeze duration is used as the countdown timer parameter for the system timer, and user instructions are registered as callback tasks to be executed after the system timer expires.

[0087] Meanwhile, considering that users might click the screen multiple times in a short period of time due to impatience, a dedicated overwrite reset logic including a continuous interruption counter was designed to prevent instructions from being suspended indefinitely. During the system timer's countdown, the system continuously listens to the terminal screen input port. To prevent malicious or impatient high-frequency clicks from causing the system to be stuck in a reset-wait loop, a continuous interruption counter is maintained in system memory, initialized to zero, with a threshold value set.

[0088] If the user does not generate any new commands during the countdown, the system will automatically execute the user's current command when the countdown ends. After receiving the command, the graphics processor will immediately start loading the video memory and outputting the corresponding 3D model asset.

[0089] If the user generates a new command, the system immediately triggers a continuous interruption counter, incrementing the counter by 1 with each new command. If the accumulated count of the continuous interruption counter reaches a threshold, the system determines that the user is in a pressing state, and in this case, ensuring immediate response takes precedence over everything else. The system immediately ignores any subsequent new commands from the user, forcibly terminates the current countdown, resets the continuous interruption counter to zero, and immediately executes the user's current command, forcibly completing the real-time output of the screen.

[0090] If the cumulative number of consecutive interruptions is less than the threshold, the system determines that the user has made a normal intention change. The system immediately sends a destruction command to the current system timer, directly destroying the user's current command that has not yet expired. Subsequently, the system uses the interaction trigger timestamp corresponding to the latest user command to re-extract data, calculate the new optimal command freeze duration, execute the latest user command, and begin the next control loop.

[0091] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0092] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A music on-demand system based on an interactive virtual scene, characterized in that, The system includes: The data acquisition module is used to obtain the corresponding accompaniment rebeat time sequence, task backlog time, graphics processor lag index, and estimated asset hard disk loading time based on the interaction trigger timestamps of user-generated commands; it also presets multiple command freeze durations. The objective function acquisition module is used to obtain the expected output time of the visual image by combining the task backlog time and the expected time of loading the asset hard disk for any instruction freeze duration; construct a one-dimensional Gaussian probability density function and perform integration operation based on the expected output time of the visual image and the graphics processor stuttering index to obtain the cumulative probability distribution of the visual output time; and obtain the step cumulative probability distribution based on the accompaniment rebeat time sequence and the expected output time of the visual image. The global cost generation module is used to obtain the audio-visual synchronization deviation cost based on the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step; to attenuate the audio-visual synchronization deviation cost based on the graphics processor stuttering index; and to obtain the global comprehensive cost based on the instruction freeze duration. The instruction execution module is used to select the optimal instruction freeze duration from all instruction freeze durations based on the global comprehensive cost value obtained for each instruction freeze duration, and then execute user instructions in conjunction with the system timer.

2. The music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, The graphics processor stuttering index includes: Based on the interaction trigger timestamp, extract the actual time consumption record of the holographic projection image of the most recently continuously output preset number of frames in the underlying state snapshot of the system at that moment; The time variance in the actual time consumption record is used as the graphics processor stuttering index.

3. A music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, The preset multiple instruction freeze durations include: Initialize an instruction freeze duration for traversal testing, a step count index for loop calculation, and a fixed step size for traversal advancement in system memory; within a preset time interval, continuously increment the instruction freeze duration according to the fixed step size to obtain multiple instruction freeze durations.

4. A music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, The method of obtaining the expected output time of the visual image by combining the freeze duration of any instruction with the backlog time of the task and the estimated time of loading the asset hard drive includes: The interaction trigger timestamp is added to the task backlog time and the current instruction freeze time, respectively. The maximum sum of the two sums is then added to the estimated time for loading the asset hard drive to obtain the expected output time of the visual image.

5. A music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, The step cumulative probability distribution obtained based on the accompaniment re-beat time sequence and the expected output time of the visual image includes: Calculate the absolute time difference between each stressed beat time point in the accompaniment stressed beat time point sequence and the expected output time of the visual image; extract the stressed beat time point with the smallest absolute time difference as the target accompaniment stressed beat time point. Based on the target accompaniment downbeat time point, a step cumulative probability distribution is constructed in the form of an indicator function; before the target accompaniment downbeat time point, the step cumulative probability distribution outputs 0; after the target accompaniment downbeat time point, the step cumulative probability distribution outputs 1.

6. A music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, The step of obtaining the audio-visual synchronization deviation cost based on the area difference between the visual output time cumulative probability distribution and the step cumulative probability distribution includes: Using the interaction trigger timestamp as the lower limit of integration and the preset dynamic safety timestamp as the upper limit of integration, definite integrals are performed on the cumulative probability distribution of visual output time and the cumulative probability distribution of step, to obtain the area difference between the cumulative probability distribution of visual output time and the cumulative probability distribution of step on the interval formed by the lower limit of integration and the upper limit of integration, which is used as the cost of audio-visual synchronization deviation.

7. A music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, The attenuation of the audio-visual synchronization deviation cost based on the graphics processor stuttering index includes: The rendering jitter attenuation coefficient is calculated by multiplying the graphics processor stuttering index by a preset attenuation constant and using the power of an exponential function with the natural constant as the base. The attenuation deviation value is obtained by multiplying the rendering jitter attenuation coefficient by the audio-visual synchronization deviation value.

8. A music on-demand system based on an interactive virtual scene according to claim 7, characterized in that, The process of obtaining the overall comprehensive value includes: The difference between the current command freeze duration and the preset upper limit of visually imperceptible waiting time is input into the one-sided penalty calculation model with non-negative boundary constraints, and the output result is used as the interaction delay cost of the current command freeze duration. The attenuation deviation cost and the interaction delay cost are weighted and summed based on preset delay dimension balance coefficients and area dimension balance coefficients to obtain the global comprehensive cost.

9. A music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, Among all instruction freeze durations, the optimal instruction freeze duration is selected based on the global comprehensive cost value obtained for each instruction freeze duration, including: Using a conventional array traversal comparison algorithm, the global comprehensive value with the smallest value is selected in chronological order, and the corresponding instruction freeze duration is taken as the optimal instruction freeze duration.

10. A music on-demand system based on an interactive virtual scene according to claim 1, characterized in that, The execution of user instructions in conjunction with the system timer includes: The optimal instruction freeze duration is used as the countdown time parameter for the system timer, and a countdown operation is performed. Maintain a continuous interruption counter in system memory, initialized to zero, and set a threshold for the number of interruptions; If the user does not generate any new command during the countdown, the user's current command will be executed when the countdown ends; if the user generates a new command, the system will immediately trigger the continuous interruption counter, and each new command will increment the continuous interruption counter by 1. If the cumulative number of consecutive interruptions reaches the threshold, the system determines that the user is in a rushing state, forcibly stops the current countdown, and immediately executes the user's current instruction. If the cumulative number of consecutive interruptions is less than the threshold, the system determines that the user is making a normal intention change, destroys the user's current instruction, recalculates the optimal instruction freeze time based on the latest user instruction, and executes the latest user instruction.

Images