Audio and virtual scene synchronous adaptation algorithm and system in digital broadcasting

Abstract

This invention discloses an algorithm and system for audio and virtual scene synchronization adaptation in digital broadcasting, belonging to the field of virtual digital broadcasting technology. The method includes acquiring video stream data and audio stream data, identifying camera switching frames, and determining the switching type; acquiring the rendering load data of the current virtual scene to calculate the load coefficient, predicting the delay duration of the camera switching frame based on the switching type, and calculating the compensation time of the audio stream data by using the predicted delay duration and load coefficient; acquiring the historical coordinate data and current action characteristics of the virtual character, determining whether the coordinates are lagging or actually displaced, and adjusting the spatial distance attenuation coefficient of the audio stream data based on the determination result; acquiring the camera's pitch angle change data and adjusting the sound image height offset of the audio stream data; and using the compensation time, spatial distance attenuation coefficient, and sound image height offset as setting parameters for the audio stream data. This invention ensures audiovisual synchronization through adjustments in three directions: time, spatial distance, and sound image height.

Description

Technical Field

[0001] This invention relates to the field of virtual digital studio technology, and in particular to an algorithm and system for synchronizing and adapting audio and virtual scenes in digital studios. Background Technology

[0002] With the popularization of digital technology in the field of drama and film, virtual digital studios have become the mainstream broadcasting mode. One of its core requirements is to achieve precise synchronization and adaptation between audio and virtual scenes, ensuring that the audience has an immersive experience of "audiovisual consistency". In the spatial sound effect algorithm of digital studios, the core calculations of sound image localization, distance attenuation, and sound field adaptation are audio-virtual scene synchronization and adaptation. This involves binding timestamps and the spatial coordinates of virtual characters together to match the spatial position of the screen frame at the same timestamp with the spatial position of the character at the same moment. Based on this binding relationship, the corresponding volume attenuation and the left and right height positions of the sound are calculated. Once a sudden delay occurs in the screen frame due to camera switching, the synchronization relationship of the timestamps is broken. The algorithm will interpret the temporal lag error as a change in spatial distance or position, thereby triggering problems such as incorrect calculation of near-field attenuation coefficient and misalignment of sound image localization.

[0003] To compensate for the above deficiencies, existing audio and video synchronization technologies mostly adopt fixed-time compensation methods, but they do not combine switching types, virtual scene rendering load, and dynamic adjustments to audio rhythm, resulting in audio timestamp misalignment. In addition, the coordinates of virtual characters read by the system are prone to lag, and existing solutions cannot effectively distinguish whether the coordinate lag is caused by delay or the actual displacement of the character. They have misjudged time delay as a change in spatial distance, resulting in incorrect adjustment of the audio spatial distance attenuation coefficient and auditory deviations such as "the character is near the field but the volume is weak" or "the character is far the field but the volume is too loud". Furthermore, when switching between tilting and panning camera positions, existing solutions do not establish a linkage between the camera tilt angle and the vertical coordinates of the character. The sound image height cannot be adjusted synchronously with the vertical position of the character, resulting in an audiovisual disconnect where "the character on screen is high up, but the sound comes from eye level".

[0004] Due to the shortcomings of existing technologies, the problem of audio and virtual scene asynchrony cannot be fundamentally solved. Therefore, there is an urgent need for an algorithm and system that is adapted to the specific scenarios of digital theatrical and film broadcasting, requires no hardware upgrades, and can achieve precise synchronization between audio and virtual scenes to overcome the deficiencies of existing technologies. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that the traditional audio-virtual scene synchronization adaptation uses fixed time compensation, which cannot fit the actual application situation. It provides an algorithm and system for audio-virtual scene synchronization adaptation in digital broadcasting, which has the advantages of ensuring accurate audio-scene adaptation by using time compensation, spatial distance attenuation adjustment and sound image height offset adjustment.

[0006] Firstly, to achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The algorithm for synchronizing and adapting audio with virtual scenes in digital broadcasting includes the following steps:

[0008] Acquire video stream data and audio stream data, perform frame analysis on the video stream data to identify camera switching frames, and determine the switching type based on the camera switching frames;

[0009] Obtain the rendering load data of the current virtual scene, calculate the load coefficient, predict the delay duration of the camera switching frame based on the switching type, and calculate the compensation time of the audio stream data using the predicted delay duration and the load coefficient.

[0010] Acquire historical coordinate data and current action characteristics of the virtual character, determine whether the current position of the virtual character is lagging or real displacement, and adjust the spatial distance attenuation coefficient of the audio stream data according to the judgment result;

[0011] Acquire the pitch angle change data of the camera position, predict the vertical coordinate change of the virtual character based on the predicted delay duration, and adjust the sound image height offset of the audio stream data based on the vertical coordinate change.

[0012] The compensation time, the spatial distance attenuation coefficient, and the sound image height offset are used as setting parameters for the audio stream data, and the audio stream data is played according to the setting parameters.

[0013] Preferably, the method for analyzing the video stream data to identify camera switching frames includes:

[0014] A preset switching feature library is provided, which stores the feature thresholds corresponding to different switching types for each bit.

[0015] Extract feature variables from consecutive frames in the video stream. The feature variables include model texture change, camera angle change rate, and character coordinate change amplitude.

[0016] The extracted feature variables are matched with the switched feature library;

[0017] Traverse the switching feature library. If at least two sets of variables in the current feature variables exceed the feature threshold, it is determined to be a camera switching frame, and the corresponding switching type is marked.

[0018] Preferably, the method for predicting the delay duration of the camera handover frame based on the handover type includes:

[0019] Construct a delay prediction model, the expression of which is as follows:

[0020] ;

[0021] ;

[0022] in, For the predicted delay duration, This is the baseline delay value corresponding to the current switching type. The baseline delay value is a preset fixed value. It's CPU utilization. This represents the total number of renderable virtual characters in the current scene, where a, b, and c are preset coefficients. The correction coefficient is determined based on the rhythmic characteristics of the audio stream data. The correction coefficient is obtained by extracting the speech rate and pause interval of the dialogue in the audio stream data as audio features and matching them with an audio library. The audio library stores the correction coefficient values ​​corresponding to the audio features.

[0023] Preferably, the method for determining whether the current virtual character's position is due to coordinate lag or actual displacement includes:

[0024] Calculate the time difference between the current time of the virtual character and the previous coordinate time, where the previous coordinate is the coordinate of the virtual character before the camera position was switched;

[0025] Calculate the theoretical maximum movement distance within the time difference based on the speed limit of the virtual character's current action characteristics;

[0026] Calculate the difference between the actual distance between the current virtual character's coordinates and the previous coordinates;

[0027] If the actual distance difference does not exceed the theoretical maximum movement distance, it is determined to be coordinate lag; otherwise, it is determined to be true displacement.

[0028] Preferably, the method for adjusting the spatial distance attenuation coefficient of the audio stream data based on the judgment result includes:

[0029] When coordinate lag is determined, the initial attenuation coefficient calculated based on the previous coordinate is corrected by time compensation according to the predicted delay duration. The time compensation correction method is as follows:

[0030] Calculate the initial attenuation coefficient based on the previous coordinate. The calculation formula is as follows:

[0031] ;

[0032] in, This is the preset value for the reference gain. This refers to the spatial distance between the virtual character and the camera position based on the previous coordinate system.

[0033] Based on the predicted delay duration Calculate the time compensation coefficient The calculation formula is as follows:

[0034] ;

[0035] in, For the preset range of time compensation coefficient values, [ [1.0-1.9];

[0036] Calculate the spatial distance attenuation coefficient after time compensation correction. The calculation formula is as follows:

[0037] ;

[0038] When the displacement is determined to be real, the initial attenuation coefficient of the previous coordinate is obtained. The initial attenuation coefficient is spatially proportionally corrected. The spatial proportional correction method is as follows:

[0039] Calculate the spatial distance between the virtual character's current coordinates and the current camera position as the real coordinate distance. The spatial distance between the previous coordinate of the virtual character and the current camera position is calculated as the previous coordinate distance. Calculate the spatial scale coefficient based on the actual coordinate distance and the previous coordinate distance. The calculation formula is:

[0040] ;

[0041] Calculate the spatial distance attenuation coefficient after spatial scale correction. The calculation formula is as follows:

[0042] .

[0043] Preferably, the method for adjusting the sound image height offset of the audio stream data according to the change in the vertical coordinate includes:

[0044] Establish a mapping relationship between the camera's pitch angle and the character's vertical coordinates. This mapping relationship adopts a linear mapping model, and the model expression is as follows:

[0045] ;

[0046] in, Predicted vertical coordinates for the character. The pitch angle of the aircraft position. The preset mapping coefficients, This is the coordinate offset constant;

[0047] The formula for calculating the rate of change of pitch angle is as follows:

[0048] ;

[0049] Based on the predicted delay duration Calculate the character's true vertical coordinates at the current moment using the following formula:

[0050] ;

[0051] in, The current vertical coordinates of the character. The vertical coordinates of the character system under the case of frame delay rendering during camera position switching. The rate of change of the vertical coordinate with the pitch angle;

[0052] The difference between the character's actual vertical coordinates and the character's system vertical coordinates at the current moment is calculated using the following formula:

[0053] ;

[0054] Based on this difference, a sound image height correction parameter is generated, using the following formula:

[0055] ;

[0056] in, For audio-visual height correction parameters, This is the preset sound-image height mapping coefficient.

[0057] Preferably, the method further includes recording the deviation between the predicted delay duration and the actual delay duration of each camera switching frame as a cumulative error, and interpolating the audio sampling points based on the cumulative error within the silent interval of the audio stream data to determine the silent interval of the current audio stream data, including:

[0058] The system monitors the energy value of the audio stream data. When the energy value is lower than the preset silence threshold and the duration reaches the preset duration, it is determined to be a silence zone.

[0059] Secondly, to achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0060] A digital broadcasting audio-virtual scene synchronization and adaptation system includes:

[0061] A frame switching identification unit is used to acquire video stream data and audio stream data, perform frame analysis on the video stream data to identify camera switching frames, and determine the switching type based on the camera switching frames;

[0062] The time compensation calculation unit is used to obtain the rendering load data of the current virtual scene, calculate the load coefficient, predict the delay duration of the camera switching frame according to the switching type, and calculate the compensation time of the audio stream data through the predicted delay duration and the load coefficient.

[0063] The spatial distance attenuation calculation unit is used to acquire the historical coordinate data and current action characteristics of the virtual character, determine whether the current position of the virtual character is coordinate lag or real displacement, and adjust the spatial distance attenuation coefficient of the audio stream data according to the judgment result.

[0064] The audio-visual height offset calculation unit is used to acquire the pitch angle change data of the camera position, predict the vertical coordinate change of the virtual character based on the predicted delay time, and adjust the audio-visual height offset of the audio stream data based on the vertical coordinate change.

[0065] The synchronization adaptation unit is used to use the compensation time, the spatial distance attenuation coefficient, and the sound image height offset as setting parameters for the audio stream data, and play the audio stream data according to the setting parameters.

[0066] Preferably, the method for predicting the delay duration of the camera handover frame based on the handover type in the time compensation calculation unit includes:

[0067] Construct a delay prediction model, the expression of which is as follows:

[0068] ;

[0069] ;

[0070] in, For the predicted delay duration, This is the baseline delay value corresponding to the current switching type. The baseline delay value is a preset fixed value. It's CPU utilization. This represents the total number of renderable virtual characters in the current scene, where a, b, and c are preset coefficients. The correction coefficient is determined based on the rhythmic characteristics of the audio stream data. The correction coefficient is obtained by extracting the speech rate and pause interval of the dialogue in the audio stream data as audio features and matching them with an audio library. The audio library stores the correction coefficient values ​​corresponding to the audio features.

[0071] Thirdly, to achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0072] A storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio-virtual scene synchronization and adaptation algorithm for digital broadcasting as described in the first aspect.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] This invention identifies camera switching frames and predicts the delay duration based on the switching type, thereby calculating the accurate compensation time for the audio stream data. This avoids time misalignment causing delays to be misjudged as changes in spatial distance, which would affect the synchronization of subsequent audio and scenes. Furthermore, by combining spatial distance attenuation and sound image height offset adjustment, the audio stream data is adjusted in three directions to make the synchronization of audio and scenes more accurate. Attached Figure Description

[0075] Figure 1 This is a flowchart of the audio-virtual scene synchronization and adaptation algorithm in digital broadcasting according to the present invention.

[0076] Figure 2 This is a schematic diagram of the audio and virtual scene synchronization and adaptation system in the digital broadcasting of the present invention. Detailed Implementation

[0077] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0078] Example 1

[0079] like Figure 1 As shown, the audio-virtual scene synchronization and adaptation algorithm in digital broadcasting includes the following steps:

[0080] Step S1: Acquire video stream data and audio stream data; perform frame analysis on the video stream data to identify camera switching frames; determine the switching type based on the camera switching frames. This step determines the switching type by accurately capturing virtual machine camera switching behavior, providing a basis for subsequent latency prediction and time compensation. The reason for determining the switching type is that different switching types have different rendering complexities and corresponding baseline latency values. Only by clearly defining the switching type can accurate prediction of latency duration be achieved.

[0081] The method for analyzing the video stream data to identify camera switching frames includes:

[0082] A preset switching feature library is provided, which stores the feature thresholds corresponding to different switching types for machine positions. The switching feature library is a pre-stored feature threshold library for different switching types in the system. Common switching types in digital broadcasting are entered. Taking theatrical broadcasting scenes as an example, such as close-up switching during the climax of dialogue, wide shot to medium shot switching during character movement, and tilt switching during emotional transitions, etc. The feature thresholds corresponding to the switching types are: model texture change ≥ 80%, camera angle change rate ≤ 10° / frame, and character coordinate change range ≤ 0.1m; the feature thresholds corresponding to tilt switching are: model texture change ≤ 30%, camera angle change rate ≥ 5° / frame, and character coordinate change range ≤ 0.05m.

[0083] Feature variables are extracted from consecutive frames in a video stream. These feature variables include model texture change, camera angle change rate, and character coordinate change amplitude. Through video stream analysis, model texture change, camera angle change rate, and character coordinate change amplitude can be extracted from two consecutive video frames.

[0084] The extracted feature variables are matched with the switched feature library;

[0085] The switching feature library is traversed. When at least two sets of variables in the current feature variables exceed the feature threshold, it is determined to be a camera switching frame, and the corresponding switching type is marked. The extracted feature variables are matched one by one with the thresholds in the switching feature library. For example, in this embodiment, the extracted feature variables are: model texture change amount 85%, lens angle change rate 8° / frame, and character coordinate change amplitude 0.08m. After traversing all switching types, it is found that the model texture change amount and character coordinate change amplitude in the current feature variables exceed the corresponding thresholds for close-up switching, and no other switching type meets the condition that at least two sets of variables exceed the threshold. Therefore, it is determined to be a camera switching frame, and the switching type is marked as close-up switching.

[0086] Step S2: Obtain the rendering load data of the current virtual scene and calculate the load coefficient. Predict the delay duration of the camera switching frame based on the switching type. Calculate the compensation time for the audio stream data using the predicted delay duration and the load coefficient. This step is used to solve the audio time misalignment problem caused by the camera switching rendering delay. By dynamically calculating the compensation time, the audio and the delayed camera switching frame are synchronized in time, providing a precise time reference for subsequent spatial adaptation and audio-visual adjustment.

[0087] The method for predicting the delay duration of the station handover frame based on the handover type includes:

[0088] Construct a delay prediction model, the expression of which is as follows:

[0089] ;

[0090] ;

[0091] in, For the predicted delay duration, This is the baseline delay value corresponding to the current switching type. The baseline delay value is a preset fixed value. It's CPU utilization. This represents the total number of renderable virtual characters in the current scene, where a, b, and c are preset coefficients. The correction coefficient is determined based on the rhythmic characteristics of the audio stream data. The correction coefficient is obtained by extracting the speech rate and pause interval of the dialogue in the audio stream data as audio features and matching them with an audio library. The audio library stores the correction coefficient values ​​corresponding to the audio features.

[0092] The following uses a specific digital broadcast program as an example to describe in detail how compensation time, spatial distance attenuation coefficient, and sound-image height offset are calculated. The system contains two virtual characters, Character A and Character B. The virtual machine supports close-up switching as a switching type, and the corresponding baseline delay value is... =60ms, the virtual scene rendering load dynamically changes with the complexity of the character's actions, CPU utilization Dcpu=60%, the audio stream is the character's dialogue audio, the speech rate is 2.5 words / second, the pause interval is 0.2s, there is no obvious background noise, the preset parameters are as follows: a=1.0, b=0.02, c=0.05, base gain G0=1.0, time compensation coefficient A=1.0, pitch angle-vertical coordinate mapping coefficient m=0.02 m / degree, coordinate offset constant d=1.0 m, sound image height mapping coefficient Kpitch=1.0, mute threshold=-55dB, mute duration threshold=0.2s, first frequency threshold=2 times / second, second frequency threshold=0.5 times / second.

[0093] Obtain the rendering load data of the current virtual scene, where the CPU utilization Dcpu = 60% and the current number of virtual characters Pn = 2. Substitute these values ​​into the formula. The calculation yields: =1.0+(60%−50%)×0.02+(2−3)×0.05=1.0+0.002−0.05=0.952;

[0094] Determine the correction factor The system extracts the speech rate of the dialogue from the audio stream (2.5 words / second, 0.2 seconds pause interval) and matches it against the system's preset audio library. The audio library contains correction coefficients corresponding to the parameters "speech rate 2-3 words / second, pause interval 0.1-0.3 seconds". =1.0;

[0095] Calculate the predicted delay duration Substitute into the delayed prediction model The calculation yields: =60ms × 0.952 + 1.0 = 57.12ms + 1.0 = 58.12ms, the compensation time is 58.12ms. This compensation time is used to adjust the timestamp of the audio stream data, achieving time synchronization between the audio and the delayed camera switching frame. The delay duration is... It is used not only to calculate the compensation time, but also to calculate the spatial distance attenuation coefficient and estimate the true vertical coordinates of the character in subsequent steps S3 and S4.

[0096] Step S3: Obtain the historical coordinate data and current action characteristics of the virtual character, determine whether the current position of the virtual character is due to coordinate lag or actual displacement, and adjust the spatial distance attenuation coefficient of the audio stream data according to the determination result. This step is used to solve the problem that the delay caused by the camera switching frame causes the coordinate lag to be misjudged as the actual displacement of the character. By distinguishing between coordinate lag and actual displacement, the spatial distance attenuation coefficient is dynamically adjusted to ensure that the audio volume matches the actual spatial position of the character accurately and avoid auditory deviation. This step is divided into two steps: the first step is to determine whether it is due to coordinate lag or actual displacement; the second step is to adjust the spatial distance attenuation coefficient. The principle of the two steps will be explained in detail below.

[0097] The method for determining whether the current virtual character's position is due to coordinate lag or actual displacement includes:

[0098] Calculate the time difference between the current time of the virtual character and the previous coordinate time, where the previous coordinate is the coordinate of the virtual character before the camera position change. Taking the above example, obtain the current time of the virtual character A, assuming it is t1=1000ms and the previous coordinate time t0=941.88ms. The time difference is Δt=t1−t0=58.12ms, which is consistent with the predicted delay time in step S2, indicating that the predicted delay time is consistent with the actual situation.

[0099] Based on the speed limit of the current action characteristics of the virtual character, calculate the theoretical maximum movement distance within the time difference; the current action characteristic of character A is small steps, and the preset speed limit of small steps is 0.1m / s. The theoretical maximum movement distance is calculated as: speed limit × time difference = 0.1m / s × 0.05812s ≈ 0.005812m.

[0100] Calculate the actual distance difference between the current virtual character's coordinates and the previous coordinates; the actual spatial distance difference between the currently read character A's coordinates and the previous coordinates is Δd = 0.004m;

[0101] If the actual distance difference does not exceed the theoretical maximum movement distance, it is determined to be coordinate lag; otherwise, it is determined to be actual displacement. Since the actual distance difference of 0.004m ≤ theoretical maximum movement distance of 0.005812m, it is determined to be coordinate lag, that is, the coordinates of the currently read virtual character are the lagging coordinates caused by the camera switching delay, and are not the actual displacement of the character.

[0102] After determining that the problem is due to coordinate lag, the method for adjusting the spatial distance attenuation coefficient of the audio stream data based on the determination result includes:

[0103] When coordinate lag is determined, the initial attenuation coefficient calculated based on the previous coordinate is corrected by time compensation according to the predicted delay duration. The time compensation correction method is as follows:

[0104] Calculate the initial attenuation coefficient based on the previous coordinate. The calculation formula is as follows:

[0105] ;

[0106] in, This is the preset value for the reference gain. This is the spatial distance between the virtual character and the camera position based on the previous coordinate; the reference gain G0 = 1.0, assuming the spatial distance between character A and the current camera position is calculated based on the previous coordinate. =1.2m, then substituting into the formula, we get: =1.0×1 / (1+1.2×0.5)=1.0 / 1.6=0.625;

[0107] Based on the predicted delay duration Calculate the time compensation coefficient The calculation formula is as follows:

[0108] ;

[0109] in, For the preset range of time compensation coefficient values, [ 1.0-1.9]; In step S2, calculate =58.12ms, and with A=1.0, substituting into the formula, we get: =1.0 + 58.12 / 100 = 1.5812;

[0110] Calculate the spatial distance attenuation coefficient after time compensation correction. The calculation formula is as follows:

[0111] ;

[0112] Substituting into the formula, we get: =0.625×1.5812≈0.988, the final spatial distance attenuation coefficient Input 0.988 to the audio rendering engine to complete the time compensation correction of the audio spatial distance attenuation coefficient, avoiding volume attenuation errors caused by coordinate lag.

[0113] The above is the spatial distance attenuation coefficient calculated for coordinate lag. It is only applicable to coordinate lag and cannot be used for actual displacement. Therefore, to avoid inaccurate adjustments due to the universality of the spatial distance attenuation coefficient, it is necessary to calculate the spatial distance attenuation coefficient again for actual displacement. When it is determined to be actual displacement, the initial attenuation coefficient of the previous coordinate is obtained. The initial attenuation coefficient is spatially proportionally corrected. The spatial proportional correction method is as follows:

[0114] Calculate the spatial distance between the virtual character's current coordinates and the current camera position as the real coordinate distance. The spatial distance between the previous coordinate of the virtual character and the current camera position is calculated as the previous coordinate distance. Calculate the spatial scale coefficient based on the actual coordinate distance and the previous coordinate distance. The calculation formula is:

[0115] ;

[0116] Calculate the spatial distance attenuation coefficient after spatial scale correction. The calculation formula is as follows:

[0117] .

[0118] Spatial distance attenuation coefficient under actual displacement conditions Spatial distance attenuation coefficient under coordinate lag condition The calculations are similar. Since this embodiment mainly addresses the technical problem of insufficient accuracy in spatial distance attenuation adjustment due to misjudging coordinate lag as actual displacement, the actual displacement will not be discussed in detail here.

[0119] Step S4: Obtain the camera's pitch angle change data, predict the virtual character's vertical coordinate change based on the predicted delay duration, and adjust the audio stream data's sound image height offset based on the vertical coordinate change. This step is used to solve the audiovisual disconnect problem caused by the mismatch between the sound image height and the character's vertical position and the camera's pitch angle when switching between tilt and pan camera positions. By linking the pitch angle with the character's vertical coordinate, the sound image height offset is corrected to ensure accurate synchronization between the sound image height and the visual presentation of the screen.

[0120] The method for adjusting the sound image height offset of the audio stream data based on the vertical coordinate change includes:

[0121] Establish a mapping relationship between the camera's pitch angle and the character's vertical coordinates. This mapping relationship adopts a linear mapping model, and the model expression is as follows:

[0122] ;

[0123] in, Predicted vertical coordinates for the character. The pitch angle of the aircraft position. The preset mapping coefficients, This is the coordinate offset constant; the purpose of this mapping relationship is to convert the change in the camera's pitch angle into a predicted value of the character's vertical coordinates, providing a basis for subsequent calculation of the actual vertical coordinates. Assuming... =0.02 m / degree, d=1.0 m, obtain the pitch angle change data of the aircraft position, current pitch angle =25°, substituting into the formula, we get: =0.02 25 + 1 = 1.5m; simultaneously calculate the pitch angle of the previous frame. Corresponding vertical coordinate prediction values: m, then =1.5-1.4=0.1m.

[0124] The formula for calculating the rate of change of pitch angle is as follows:

[0125] ;

[0126] Current pitch angle =25°, pitch angle of the previous frame The time difference Δt = 58.12 ms, substituting into the formula, we get: Pitch angle change rate = (25°−20°) / 0.05812s≈86.03° / s;

[0127] Based on the predicted delay duration Calculate the character's true vertical coordinates at the current moment using the following formula:

[0128] ;

[0129] in, The current vertical coordinates of the character. The vertical coordinates of the character system under the case of frame delay rendering during camera position switching. The rate of change of the vertical coordinates with the pitch angle; the system reads the vertical coordinates of the character system. =1.4m, rate of change of vertical coordinate with pitch angle =0.1 / 0.05812≈1.7206 m / s, substitute into the formula =1.4m + 1.7206 m / s × 0.05812s ≈ 1.4m + 0.1000m = 1.5m. The calculated result is consistent with the predicted value of the character's vertical coordinates. Consistent;

[0130] The difference between the character's actual vertical coordinates and the character's system vertical coordinates at the current moment is calculated using the following formula:

[0131] =1.5-1.4=0.1m;

[0132] Based on this difference, a sound image height correction parameter is generated, using the following formula:

[0133] ;

[0134] in, For audio-visual height correction parameters, Assuming the preset sound image height mapping coefficient, =1, then substitute into the formula 0.1 1 = 0.1m, this parameter This is the sound image height offset, used to adjust the vertical position of the audio sound image.

[0135] Step S5: The compensation time, the spatial distance attenuation coefficient, and the sound image height offset are used as setting parameters for the audio stream data. The audio stream data is played according to the setting parameters. This step is used to integrate the three core setting parameters generated in the previous steps and input them into the audio rendering engine to control the audio to play according to the requirements of time synchronization, volume adaptation, and sound image accuracy, so as to achieve complete synchronization and adaptation between the audio and the virtual scene.

[0136] To eliminate the accumulation of camera cut delay deviations, prevent errors from escalating with switching frequency, and ensure that audio and virtual scenes remain synchronized throughout long-term broadcasts, thereby improving broadcast stability, this method also includes recording the deviation between the predicted and actual delay durations of each camera cut frame as a cumulative error. Based on this cumulative error, audio sampling points are interpolated within the silent intervals of the audio stream data to determine the current silent interval of the audio stream data.

[0137] The system monitors the energy value of the audio stream data. When the energy value is lower than the preset silence threshold and the duration reaches the preset duration, it is determined to be a silence zone.

[0138] For example, after each camera angle change and frame rendering, the actual delay is calculated to be 59ms, and the predicted delay is 58.12ms. The difference between the two is 0.88ms. This difference is added to the cumulative error, and an error record table is established to track it in real time. The energy value of the audio stream data is monitored. When the detected energy value is -58dB, which is lower than the silence threshold of -55dB and the duration is 0.25s, which reaches the duration threshold of 0.2s, it is determined to be a silent interval. Within this silent interval, the virtual character is in the pause between lines. Then, a temporal stretching algorithm is used to interpolate the audio sampling points. When inserting, the effective sampling points adjacent to the silent interval in the current audio stream data are used as the reference. A transitional sampling point is generated through a linear interpolation algorithm to ensure that the amplitude and phase of the inserted point are seamlessly connected with the original sampling point, avoiding noise and sound breaks, and not destroying the integrity of the lines and background sounds.

[0139] Example 2

[0140] like Figure 2As shown, the audio-virtual scene synchronization and adaptation system in digital broadcasting includes:

[0141] A frame switching identification unit is used to acquire video stream data and audio stream data, perform frame analysis on the video stream data to identify camera switching frames, and determine the switching type based on the camera switching frames;

[0142] The time compensation calculation unit is used to obtain the rendering load data of the current virtual scene, calculate the load coefficient, predict the delay duration of the camera switching frame according to the switching type, and calculate the compensation time of the audio stream data through the predicted delay duration and the load coefficient.

[0143] The spatial distance attenuation calculation unit is used to acquire the historical coordinate data and current action characteristics of the virtual character, determine whether the current position of the virtual character is coordinate lag or real displacement, and adjust the spatial distance attenuation coefficient of the audio stream data according to the judgment result.

[0144] The audio-visual height offset calculation unit is used to acquire the pitch angle change data of the camera position, predict the vertical coordinate change of the virtual character based on the predicted delay time, and adjust the audio-visual height offset of the audio stream data based on the vertical coordinate change.

[0145] The synchronization adaptation unit is used to use the compensation time, the spatial distance attenuation coefficient, and the sound image height offset as setting parameters for the audio stream data, and play the audio stream data according to the setting parameters.

[0146] The method for predicting the delay duration of the camera switching frame based on the switching type in the time compensation calculation unit includes:

[0147] Construct a delay prediction model, the expression of which is as follows:

[0148] ;

[0149] ;

[0150] in, For the predicted delay duration, This is the baseline delay value corresponding to the current switching type. The baseline delay value is a preset fixed value. It's CPU utilization. This represents the total number of renderable virtual characters in the current scene, where a, b, and c are preset coefficients. The correction coefficient is determined based on the rhythmic characteristics of the audio stream data. The correction coefficient is obtained by extracting the speech rate and pause interval of the dialogue in the audio stream data as audio features and matching them with an audio library. The audio library stores the correction coefficient values ​​corresponding to the audio features.

[0151] Since Embodiment 1 and Embodiment 2 are essentially the same, the units in Embodiment 2 will not be explained in detail.

[0152] The present invention also discloses a storage medium storing a computer program, which, when executed by a processor, implements the audio-virtual scene synchronization and adaptation algorithm in digital broadcasting as described in Embodiment 1.

[0153] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0154] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An algorithm for synchronizing and adapting audio and virtual scenes in digital broadcasting, characterized in that, Includes the following steps: Acquire video stream data and audio stream data, perform frame analysis on the video stream data to identify camera switching frames, and determine the switching type based on the camera switching frames; Obtain the rendering load data of the current virtual scene, calculate the load coefficient, predict the delay duration of the camera switching frame based on the switching type, and calculate the compensation time of the audio stream data using the predicted delay duration and the load coefficient. Acquire historical coordinate data and current action characteristics of the virtual character, determine whether the current position of the virtual character is lagging or real displacement, and adjust the spatial distance attenuation coefficient of the audio stream data according to the judgment result; Acquire the pitch angle change data of the camera position, predict the vertical coordinate change of the virtual character based on the predicted delay duration, and adjust the sound image height offset of the audio stream data based on the vertical coordinate change. The compensation time, the spatial distance attenuation coefficient, and the sound image height offset are used as setting parameters for the audio stream data, and the audio stream data is played according to the setting parameters.

2. The audio-virtual scene synchronization and adaptation algorithm in digital broadcasting according to claim 1, characterized in that, The method for analyzing the video stream data to identify camera switching frames includes: A preset switching feature library is provided, which stores the feature thresholds corresponding to different switching types for each bit. Extract feature variables from consecutive frames in the video stream. The feature variables include model texture change, camera angle change rate, and character coordinate change amplitude. The extracted feature variables are matched with the switched feature library; Traverse the switching feature library. If at least two sets of variables in the current feature variables exceed the feature threshold, it is determined to be a camera switching frame, and the corresponding switching type is marked.

3. The audio-virtual scene synchronization and adaptation algorithm in digital broadcasting according to claim 1, characterized in that, The method for predicting the delay duration of the station handover frame based on the handover type includes: Construct a delay prediction model, the expression of which is as follows: ； ； in, For the predicted delay duration, This is the baseline delay value corresponding to the current switching type. The baseline delay value is a preset fixed value. It's CPU utilization. This represents the total number of renderable virtual characters in the current scene, where a, b, and c are preset coefficients. The correction coefficient is determined based on the rhythmic characteristics of the audio stream data. The correction coefficient is obtained by extracting the speech rate and pause interval of the dialogue in the audio stream data as audio features and matching them with an audio library. The audio library stores the correction coefficient values ​​corresponding to the audio features.

4. The audio-virtual scene synchronization and adaptation algorithm in digital broadcasting according to claim 1, characterized in that, The method for determining whether the current virtual character's position is due to coordinate lag or actual displacement includes: Calculate the time difference between the current time of the virtual character and the previous coordinate time, where the previous coordinate is the coordinate of the virtual character before the camera position was switched; Calculate the theoretical maximum movement distance within the time difference based on the speed limit of the virtual character's current action characteristics; Calculate the difference between the actual distance between the current virtual character's coordinates and the previous coordinates; If the actual distance difference does not exceed the theoretical maximum movement distance, it is determined to be coordinate lag; otherwise, it is determined to be true displacement.

5. The audio-virtual scene synchronization and adaptation algorithm in digital broadcasting according to claim 4, characterized in that, The method for adjusting the spatial distance attenuation coefficient of audio stream data based on the judgment result includes: When coordinate lag is determined, the initial attenuation coefficient calculated based on the previous coordinate is corrected by time compensation according to the predicted delay duration. The time compensation correction method is as follows: Calculate the initial attenuation coefficient based on the previous coordinate. The calculation formula is as follows: ； in, This is the preset value for the reference gain. This refers to the spatial distance between the virtual character and the camera position based on the previous coordinate system. Based on the predicted delay duration Calculate the time compensation coefficient The calculation formula is as follows: ； in, For the preset range of time compensation coefficient values, [ [1.0-1.9]; Calculate the spatial distance attenuation coefficient after time compensation correction. The calculation formula is as follows: ； When the displacement is determined to be real, the initial attenuation coefficient of the previous coordinate is obtained. The initial attenuation coefficient is spatially proportionally corrected. The spatial proportional correction method is as follows: Calculate the spatial distance between the virtual character's current coordinates and the current camera position as the real coordinate distance. The spatial distance between the previous coordinate of the virtual character and the current camera position is calculated as the previous coordinate distance. Calculate the spatial scale coefficient based on the actual coordinate distance and the previous coordinate distance. The calculation formula is: ； Calculate the spatial distance attenuation coefficient after spatial scale correction. The calculation formula is as follows: 。 6. The audio-virtual scene synchronization and adaptation algorithm in digital broadcasting according to claim 1, characterized in that, The method for adjusting the sound image height offset of the audio stream data based on the vertical coordinate change includes: Establish a mapping relationship between the camera's pitch angle and the character's vertical coordinates. This mapping relationship adopts a linear mapping model, and the model expression is as follows: ； in, Predicted vertical coordinates for the character. The pitch angle of the aircraft position. The preset mapping coefficients, This is the coordinate offset constant; The formula for calculating the rate of change of pitch angle is as follows: ； Based on the predicted delay duration Calculate the character's true vertical coordinates at the current moment using the following formula: ； in, The current vertical coordinates of the character. The vertical coordinates of the character system under the case of frame delay rendering during camera position switching. The rate of change of the vertical coordinate with the pitch angle; The difference between the character's actual vertical coordinates and the character's system vertical coordinates at the current moment is calculated using the following formula: ； Based on this difference, a sound image height correction parameter is generated, using the following formula: ； in, For audio-visual height correction parameters, This is the preset sound-image height mapping coefficient.

7. The audio-virtual scene synchronization and adaptation algorithm in digital broadcasting according to claim 1, characterized in that, The method also includes recording the deviation between the predicted delay time and the actual delay time of each camera switching frame as a cumulative error, and interpolating the audio sampling points based on the cumulative error within the silent interval of the audio stream data to determine the silent interval of the current audio stream data. The system monitors the energy value of the audio stream data. When the energy value is lower than the preset silence threshold and the duration reaches the preset duration, it is determined to be a silence zone.

8. A system for synchronized adaptation of audio and virtual scenes in digital broadcasting, characterized in that, include: A frame switching identification unit is used to acquire video stream data and audio stream data, perform frame analysis on the video stream data to identify camera switching frames, and determine the switching type based on the camera switching frames; The time compensation calculation unit is used to obtain the rendering load data of the current virtual scene, calculate the load coefficient, predict the delay duration of the camera switching frame according to the switching type, and calculate the compensation time of the audio stream data through the predicted delay duration and the load coefficient. The spatial distance attenuation calculation unit is used to acquire the historical coordinate data and current action characteristics of the virtual character, determine whether the current position of the virtual character is coordinate lag or real displacement, and adjust the spatial distance attenuation coefficient of the audio stream data according to the judgment result. The audio-visual height offset calculation unit is used to acquire the pitch angle change data of the camera position, predict the vertical coordinate change of the virtual character based on the predicted delay time, and adjust the audio-visual height offset of the audio stream data based on the vertical coordinate change. The synchronization adaptation unit is used to use the compensation time, the spatial distance attenuation coefficient, and the sound image height offset as setting parameters for the audio stream data, and play the audio stream data according to the setting parameters.

9. The audio and virtual scene synchronization and adaptation system in digital broadcasting according to claim 8, characterized in that, The method for predicting the delay duration of the camera switching frame based on the switching type in the time compensation calculation unit includes: Construct a delay prediction model, the expression of which is as follows: ； ； in, For the predicted delay duration, This is the baseline delay value corresponding to the current switching type. The baseline delay value is a preset fixed value. It's CPU utilization. This represents the total number of renderable virtual characters in the current scene, where a, b, and c are preset coefficients. The correction coefficient is determined based on the rhythmic characteristics of the audio stream data. The correction coefficient is obtained by extracting the speech rate and pause interval of the dialogue in the audio stream data as audio features and matching them with an audio library. The audio library stores the correction coefficient values ​​corresponding to the audio features.

10. A storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the audio-virtual scene synchronization and adaptation algorithm in digital broadcasting as described in any one of claims 1-7.

Images