A system and device for reducing audio and lighting synchronization delay

Abstract

The present invention discloses a system and device for reducing audio and lighting synchronization delay, comprising: an audio feature extraction module for converting an input audio signal into a Mel spectrum feature S mel ; Lighting parameter generation module, used to generate lighting control parameters V light ; Prediction module, used to use LSTM based on the Mel spectrum features S in historical data mel and lighting control parameters V light Predicting future lighting instructions C pred , LSTM includes a switchable weight matrix; a real-time processing module for processing the data based on the current Mel spectrum feature S through the FFT unit mel Generate real-time lighting instructions C real ; Arbitration module, used to compare future light instructions C pred With real-time lighting instructions C real The present invention can reduce the audio and lighting synchronization delay through all-round optimization of scene-based model training, real-time delay calibration, multi-dimensional arbitration decision-making and hardware acceleration processing, and realize real-time linkage between lighting effects and performance content.

Description

Technical Field

[0001] The present invention relates to the field of lighting control, and in particular to a system and device for reducing audio and lighting synchronization delay. Background Art

[0002] KTV lighting needs to be dynamically adjusted according to the audio content to enhance the singing atmosphere. Typical control dimensions include:

[0003] Color and brightness control: For example, lyrical songs use warm tones with low brightness to create a cozy feeling, while rock songs switch to cool tones and increase brightness to enhance the impact;

[0004] The lights should flash in sync with the rhythm. In the accompaniment section, the lights should flash at a high frequency according to the drum beat frequency, while the lights in the vocal section should turn into a soft gradient.

[0005] The spatial layout is coordinated, and the main lights, background lights, and contour lights in the box need to change in a coordinated manner according to their geometric positions.

[0006] Therefore, it is necessary to use a system to reduce the audio and lighting synchronization delay to reduce the problem of audio and lighting synchronization delay in KTV.

[0007] Existing technologies have problems such as accumulated processing chain delay, implicit delay caused by insufficient scene adaptation, and large delay fluctuations in dynamic environments. Therefore, a system and device for reducing audio and lighting synchronization delay are proposed. Summary of the Invention

[0008] The technical problem to be solved by the present invention is how to solve the problems in the existing technology such as the accumulation of processing chain delays, implicit delays caused by insufficient scene adaptation, and large delay fluctuations in dynamic environments. A system and device for reducing audio and lighting synchronization delay are provided.

[0009] The present invention solves the above-mentioned technical problems through the following technical solutions, which include:

[0010] Audio feature extraction module, used to convert the input audio signal into Mel spectrum feature S mel ;

[0011] Lighting parameter generation module, used to generate lighting control parameters V light ;

[0012] The prediction module is used to predict the Mel spectrum features S in the historical data through LSTM. mel and lighting control parameters V light Predicting future lighting instructions C pred ,LSTM contains switchable weight matrices;

[0013] Real-time processing module, used to process the current Mel spectrum feature S through the FFT unit melGenerate real-time lighting instructions C real ;

[0014] Arbitration module, used to compare future light instructions C pred With real-time lighting instructions C real When D>δ, the future lighting instruction C is output. pred Otherwise, output real-time lighting instruction C real as an arbitration order;

[0015] The joint encoding and decoding module is used to convert the Mel spectrum feature S corresponding to the arbitration instruction mel and lighting control parameters V light The joint encoding is a joint vector Z and decoded into a protocol control signal.

[0016] Furthermore, the prediction module includes:

[0017] The human voice energy ratio calculation unit is used to calculate the human voice energy ratio. The specific process is as follows:

[0018] , where E 300-3400Hz E is the energy of the audio signal in the frequency band from 300Hz to 3400Hz, which is the main frequency range of human voice. total is the total energy of the audio signal;

[0019] Weight switching unit, used in R vocal When γ is greater than γ, the first weight matrix W1 is used, otherwise the second weight matrix W2 is used.

[0020] Furthermore, the method further includes an LSTM training module, which generates a first weight matrix W1 and a second weight matrix W2 by the following steps:

[0021] Step a: Collect the audio dataset Dvocal for scenes dominated by human voice and the dataset Dmusic for scenes dominated by music;

[0022] Step b: Constructing an LSTM network architecture, wherein the network comprises an input layer, a hidden layer, an output layer, and a trainable weight matrix W;

[0023] Step c: Define the loss function: 2;

[0024] W is the weight matrix of the LSTM network; is the predicted lighting instruction of the i-th sample under weight W; is the real lighting instruction of the i-th sample; N is the number of training samples;

[0025] Step d: Use the gradient descent algorithm combined with the back propagation through time (BPTT) algorithm to iteratively optimize the weight matrix:

[0026] Calculate the gradient of the loss function with respect to the weights based on time backpropagation: ;

[0027] Update the weight matrix: , where η is the learning rate, is the loss function Gradient of the weight matrix W;

[0028] Apply the process from step a to step d to the audio dataset Dvocal for the scene dominated by human voice to optimize and obtain the optimized first weight matrix W1;

[0029] Applying steps a to d to the music-dominant scene dataset Dmusic to optimize the process, and obtaining an optimized second weight matrix W2;

[0030] The LSTM training module is deployed on a cloud server, and the first weight matrix W1 and the second weight matrix W2 are updated to the local prediction module via a wireless network.

[0031] Furthermore, the LSTM training module is further configured to perform the following process:

[0032] We added vocal feature enhancement to the Dvocal audio dataset for vocal-dominated scenes. This enhancement includes fundamental frequency detection and enhancement, and harmonic structure analysis and optimization.

[0033] Add low-frequency enhancement to the music-dominant scene dataset Dmusic. The enhancements include beat detection and rhythm pattern extraction, and low-frequency energy envelope smoothing.

[0034] The gradient is calculated using the Backpropagation Through Time (BPTT) algorithm ;

[0035] Batch normalization and Dropout techniques are applied during training to prevent overfitting.

[0036] Furthermore, the specific processing process of the joint encoding and decoding module is:

[0037] Mel spectrum feature S mel Perform time domain maximum pooling. The specific process is:

[0038] , where S mel (t, f) is the original Mel spectrum feature, where t represents the time dimension, f represents the frequency dimension, and max t To obtain the maximum value in the time dimension t;

[0039] Light control parameter V light Perform spatial weighted coding. The specific process is as follows:

[0040] ;

[0041] where w i is the spatial position weight, which is pre-calculated based on the geometric relationship of the light layout. i are the control parameters of each lighting unit;

[0042] Generate the joint vector Z. The specific process is:

[0043] , where b is the bias vector, Refers to the concatenation of the Mel spectrum feature Smel′ after the maximum pooling in the time domain and the light parameter Vlight′ after the spatial weighted encoding.

[0044] Furthermore, the system further includes a calibration module configured to receive an actual delay T measured externally. actual , dynamically adjust the prediction step size M, the specific process is:

[0045] ;

[0046] Where T target is the preset target delay value.

[0047] Furthermore, the process of obtaining the difference D is as follows:

[0048] ;

[0049] Where α is the difference index of the arbitration module, which is used to quantitatively predict the future lighting instructions C pred With real-time lighting instructions C real ; c∈{R,G,B} traverses the three color channels of red (Red), green (Green), and blue (Blue), covering the color dimension of the light; I c,pred is the predicted brightness value of channel c, such as an integer from 0 to 255, representing the brightness intensity of the light in this color channel; I c,real is the brightness value of channel c calculated in real time and output by the real-time processing module; β is the weight coefficient of frequency difference, which is used to balance the difference contribution of color and frequency dimensions, F freq,pred is the predicted light flicker frequency, F freq,real The light flashing frequency is calculated in real time.

[0050] A device for reducing audio and lighting synchronization delay, the device comprising:

[0051] Audio codec chip, used to realize audio feature extraction;

[0052] Dual-core processor for running prediction module and real-time processing module;

[0053] Training coprocessor for LSTM weight matrix generation;

[0054] Programmable logic array for generating arbitration instructions and implementing joint encoding and decoding.

[0055] Compared with the existing technology, the present invention has the following advantages: the system and device for reducing audio and lighting synchronization delay use LSTM to predict future lighting instructions, combine real-time processing results for intelligent arbitration, compensate for system processing delays in advance, and achieve synchronous response of audio and lighting. The LSTM weight matrix is ​​switched according to the proportion of human voice energy, and the lighting control of voice-dominated and music-dominated scenes is optimized respectively, thereby improving the synchronization accuracy in different scenes. The prediction step size is dynamically adjusted according to the actual measured delay, so that the system can adapt to the delay characteristics in different environments and ensure synchronization stability. The joint codec module performs time-domain pooling, spatial weighting and feature fusion on the audio mel spectrum and lighting parameters, compresses the data volume while retaining key synchronization information, and improves the efficiency of command transmission. The device uses an audio codec chip, a dual-core processor and a programmable logic array to achieve parallel processing of feature extraction, prediction calculation and arbitration coding and decoding, meets real-time requirements and supports hardware customization, making the system more worthy of promotion and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0057] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process. However, the protection scope of the present invention is not limited to the following embodiment.

[0058] like Figure 1 As shown, this embodiment provides a technical solution: a system for reducing audio and lighting synchronization delay, comprising:

[0059] Audio feature extraction module, used to convert the input audio signal into Mel spectrum feature S mel ;

[0060] Lighting parameter generation module, used to generate lighting control parameters V light ;

[0061] The prediction module is used to predict the Mel spectrum features S in the historical data through LSTM. mel and lighting control parameters V light Predicting future lighting instructions C pred,LSTM contains switchable weight matrices;

[0062] Real-time processing module, used to process the current Mel spectrum feature S through the FFT unit mel Generate real-time lighting instructions C real ;

[0063] Arbitration module, used to compare future light instructions C pred With real-time lighting instructions C real When D>δ, the future lighting instruction C is output. pred Otherwise, output real-time lighting instruction C real As an arbitration instruction, δ is the preset threshold;

[0064] The joint encoding and decoding module is used to convert the Mel spectrum feature S corresponding to the arbitration instruction mel and lighting control parameters V light The joint encoding is a joint vector Z and decoded into a protocol control signal.

[0065] The prediction module includes:

[0066] The human voice energy ratio calculation unit is used to calculate the human voice energy ratio. The specific process is as follows:

[0067] , where E 300-3400Hz E is the energy of the audio signal in the frequency band from 300Hz to 3400Hz, which is the main frequency range of human voice. total is the total energy of the audio signal;

[0068] Weight switching unit, used in R vocal When it is greater than γ, the first weight matrix W1 is used, otherwise the second weight matrix W2 is used, and γ is the threshold;

[0069] Based on the frequency characteristic difference between human voice and music, the human voice energy ratio R is calculated. vocal Switching the weight matrix enables the LSTM model to specifically learn the audio-light mapping relationship in different scenarios.

[0070] For example, when human voices dominate, the response to speech features is strengthened; when music dominates, the response to rhythm and low-frequency energy is strengthened, thereby improving the accuracy of lighting synchronization.

[0071] If a single weight matrix is ​​used to process all scenarios, the model may have fitting deviations in vocal or music scenarios.

[0072] For example, in pure music scenes, if the model over-learns the characteristics of human voice, the lighting effects will be out of sync with the rhythm. However, in this case, by switching weights, the model can focus on key features in different scenes and reduce prediction errors.

[0073] When the user sings, the energy of the 300-3400Hz frequency band in the audio is high. vocal >γ, the system switches to the first weight matrix W1. At this time, the LSTM model will predict the lighting instructions based on the pitch and strength of the human voice. For example, when the high-pitched passage is in progress, the light brightness will suddenly increase, and the color will change with the tone, such as from blue to red.

[0074] During ventilation intervals, the light brightness is expected to transition smoothly to avoid sudden flickering.

[0075] When playing electronic music, low-frequency drum beats and rhythms dominate. vocal ≤γ, the system switches to the second weight matrix W2. At this time, the LSTM model pays more attention to the low-frequency energy envelope and beat characteristics. For example, when there is a heavy bass drum beat, the light is predicted to flash at a high frequency with the rhythm, and the color tends to be purple or orange.

[0076] During the melody transition section, the predicted lighting gradient speed is synchronized with the chord changes to enhance the musical atmosphere.

[0077] The LSTM training module is further included, and the LSTM training module generates a first weight matrix W1 and a second weight matrix W2 through the following steps:

[0078] Step a: Collect the audio dataset Dvocal for scenes dominated by human voice and the dataset Dmusic for scenes dominated by music;

[0079] Step b: Constructing an LSTM network architecture, wherein the network comprises an input layer, a hidden layer, an output layer, and a trainable weight matrix W;

[0080] Step c: Define the loss function: 2;

[0081] W is the weight matrix of the LSTM network;

[0082] is the predicted lighting instruction of the i-th sample under weight W;

[0083] is the actual lighting instruction of the i-th sample;

[0084] N is the number of training samples;

[0085] Step d: Use the gradient descent algorithm combined with the back propagation through time (BPTT) algorithm to iteratively optimize the weight matrix:

[0086] Calculate the gradient of the loss function with respect to the weights based on time backpropagation: This process describes the overall optimization process of LSTM training. Calculating the gradient of the loss function with respect to the weights is the core step of the gradient descent algorithm. The essence of gradient descent is to guide the iterative update of the weight matrix by calculating the gradient of the loss function with respect to the weights to minimize the difference between the predicted value and the true value.

[0087] Update the weight matrix: , where η is the learning rate, is the loss function Gradient of the weight matrix W;

[0088] Apply the process from step a to step d to the audio dataset Dvocal for scenes dominated by human voices to optimize and obtain an optimized first weight matrix W1;

[0089] Applying steps a to d to the music-dominant scene dataset Dmusic to optimize the process, and obtaining an optimized second weight matrix W2;

[0090] The LSTM training module is deployed on a cloud server, and the first weight matrix W1 and the second weight matrix W2 are updated to the local prediction module via a wireless network;

[0091] The above process trains the weight matrices for voice-dominated and music-dominated scenarios respectively, enabling the LSTM model to deeply learn the mapping relationship between different audio features and lighting control.

[0092] For example, in a human voice scene, the first weight matrix W1 strengthens the capture of speech features such as fundamental frequency and harmonics;

[0093] In the music scene, the second weight matrix W2 strengthens the response to music features such as beats and low-frequency energy envelopes, avoiding prediction bias of a single model across scenes.

[0094] The LSTM training module is deployed on a cloud server, leveraging the server's high computing power to complete complex model iterations. This eliminates the need for local devices to consume large amounts of computing resources and adapts to the low power consumption requirements of embedded devices.

[0095] When audio scene data or user needs change, the weight matrix can be retrained in the cloud and pushed locally via wireless network, enabling the system to continuously evolve and improve synchronization without physically replacing hardware.

[0096] For example, in an online concert, there will be a mixed scene of vocals and accompaniment. In this case, the cloud server pre-trains the first weight matrix W1 and the second weight matrix W2 based on Dvocal and Dmusic respectively. When the host sings: the system detects that the vocal energy accounts for a high proportion, calls the first weight matrix W1, and the model predicts the light color switch based on the pitch change of the vocals, such as C corresponds to blue and E corresponds to yellow;

[0097] When entering the accompaniment stage, the model switches to the second weight matrix W2, and predicts the light flashing frequency according to the drum rhythm, such as a strong light per beat, to ensure the synchronization accuracy of light and audio in both scenarios.

[0098] During the initial deployment of party music playback, the local prediction module uses the default first weight matrix W1 or second weight matrix W2 trained on the cloud. If a user frequently plays electronic music, which is a music-dominated scenario, the cloud can re-optimize the second weight matrix W2 by collecting the user's playback data:

[0099] Added training for the low-frequency oscillation characteristics of synthesizers in electronic music, so that the second weight matrix W2 can predict the gradual change pattern of lights as the BPM changes;

[0100] The optimized second weight matrix W2 is pushed to the lighting controller via Wi-Fi. The next time the same music is played, the synchronization between the lighting effect and the rhythm is significantly improved without the need for manual settings by the user.

[0101] The LSTM training module is also used to perform the following process:

[0102] We added vocal feature enhancement to the Dvocal audio dataset for vocal-dominated scenes. This enhancement includes fundamental frequency detection and enhancement, and harmonic structure analysis and optimization.

[0103] Add low-frequency enhancement to the music-dominant scene dataset Dmusic. The enhancements include beat detection and rhythm pattern extraction, and low-frequency energy envelope smoothing.

[0104] Use the back propagation through time (BPTT) algorithm to expand the time series dependency of LSTM and calculate the gradient of the loss function with respect to the weight in step d. Backpropagation through time (BPTT) is a specific implementation of gradient descent in recurrent neural networks, specifically in LSTM. Since LSTM processes time series data, its hidden states have dependencies in the time dimension. Therefore, it is necessary to unfold the time series and backpropagate the gradient along the time dimension to resolve long-distance dependencies.

[0105] BPTT does not deviate from the framework of gradient descent, but refines the specific steps of gradient calculation based on the structural characteristics of LSTM.

[0106] For example, for data in scenes dominated by human voices, the back propagation through time (BPTT) algorithm will perform gradient calculations on features such as fundamental frequency and harmonics along the time dimension.

[0107] For music-dominated scenarios, BPTT calculates the gradient by combining the temporal characteristics of beats and low-frequency energy envelopes. Gradient descent is a methodology that provides the mathematical principles and iterative direction for weight optimization.

[0108] The Back Propagation Through Time (BPTT) algorithm is a tool that implements efficient calculation of gradients in the time dimension based on the temporal characteristics of LSTM.

[0109] The gradient descent framework relies on the back propagation through time (BPTT) algorithm to complete the training of the LSTM weight matrix;

[0110] Batch normalization and Dropout techniques are applied during training to prevent overfitting;

[0111] By adding fundamental frequency detection and harmonic enhancement to the vocal dataset, the LSTM model can accurately capture the singer's pitch and timbre changes, allowing the lighting effects to dynamically adjust with the vocal emotions, enhancing the immersiveness of the performance, and synchronously optimizing the accompaniment rhythm and lighting rhythm. Low-frequency enhancement is added to the music accompaniment to ensure that the lights can flash to the rhythm of low-frequency instruments such as drums and bass during the song accompaniment section, avoiding lighting delays when switching between vocals and accompaniment.

[0112] Combining the BPTT algorithm and Dropout technology, the model can adapt to the vocal range and singing style differences of different singers, reduce the lighting prediction deviation caused by individual voice characteristics, and avoid discordant phenomena such as out-of-tune lighting.

[0113] For example, when a vocal performance occurs in a KTV, the system enhances the vocal features and the LSTM model performs the following optimizations using the first weight matrix W1:

[0114] Fundamental frequency detection is linked to light color. When the model detects that the singer is singing high notes, it predicts that the light brightness will increase based on the enhanced fundamental frequency characteristics and switch to warm colors, such as red. When singing low notes, the light turns dark blue, and the color change speed is synchronized with the pitch change slope.

[0115] Harmonic structure optimization and lighting gradients: When a singer uses falsetto or vibrato, harmonic enhancement enables the model to identify high-frequency overtones and predict delicate color gradients in the lights, such as from purple to pink, avoiding the flickering chaos of the lights caused by the original feature noise.

[0116] When there is an accompaniment bridge, such as the prelude to the chorus of a song, and the accompaniment section begins, the system enhances the music features and performs the following optimization through the second weight matrix W2:

[0117] Low-frequency energy envelope synchronized with the rhythm of the lights:

[0118] When an 80Hz bass drum beat is detected in the accompaniment, the model predicts that the light will flash once per beat through the smoothed low-frequency energy envelope, and the color switches between purple and blue along with the bass melody; if the drum beat speeds up, such as the BPM increases from 120 to 140, the light flashing frequency increases synchronously without any sense of delay.

[0119] Beat pattern extraction and lighting choreography: The model recognizes the syncopated rhythm in the accompaniment, such as the off-beat drum beat, and predicts in advance the faint flickering of lights at non-beat positions to enhance the rhythm of the music. For example, the lights are bright on the beat and dim on the off-beat, creating a layered visual effect.

[0120] KTVs often face singing data from different users, and the training optimization mechanism in this case plays a role:

[0121] When users sing long phrases continuously, the model accurately calculates the gradient through time backpropagation, avoiding the lag in light prediction caused by long-distance dependencies. For example, the timing of the light dimming at the end of a long note is completely synchronized with the attenuation of the tail note.

[0122] If a user is accustomed to singing with nasal resonance, Dropout will randomly ignore some neuronal connections to prevent the model from over-learning this feature, ensuring that the lighting effects can still adapt when other users sing. For example, when different singers sing the same song, the lighting's response pattern to pitch remains consistent.

[0123] The specific processing process of the joint encoding and decoding module is as follows:

[0124] Mel spectrum feature S mel Perform time domain maximum pooling. The specific process is:

[0125] , where S mel (t, f) is the original Mel spectrum feature, where t represents the time dimension, f represents the frequency dimension, and max t To obtain the maximum value in the time dimension t;

[0126] Light control parameter V light Perform spatial weighted coding. The specific process is as follows:

[0127] ;

[0128] where w i is the spatial position weight, which is pre-calculated based on the geometric relationship of the light layout. i are the control parameters of each lighting unit;

[0129] Generate the joint vector Z. The specific process is:

[0130] , where b is the bias vector, Refers to the concatenation of the Mel spectrum feature Smel′ after the maximum pooling in the time domain and the light parameter Vlight′ after the spatial weighted coding;

[0131] By performing time-domain maximum pooling and maximizing the mel-spectrogram in the time dimension, key features such as high-frequency peaks and rhythmic mutations in human voices can be quickly captured, thus avoiding redundant data interference and improving the real-time performance of lighting response.

[0132] Spatial weighted coding pre-calculates weights based on the lighting layout of the KTV box, allowing lighting units at different positions to change collaboratively according to geometric relationships, avoiding cluttered lighting effects and enhancing spatial immersion.

[0133] Joint vector generation encodes Mel-spectrogram features and lighting parameters into a unified vector, reducing data transmission while retaining key synchronization information, ensuring accurate mapping of audio features and lighting control commands, and reducing system processing latency.

[0134] For example, when singing a climax of a vocal segment, the following optimization is performed when the user sings the high notes of the chorus:

[0135] Maximal pooling in time domain captures high-pitched features:

[0136] The maximum value of the Mel spectrum in the time dimension t corresponds to the moment of high-pitched burst. Smel′ extracts the spectrum features of this moment, and the system quickly identifies the high-pitched event.

[0137] Spatial weighted coding coordinates the lighting layout. Assume that the lighting layout of a KTV box is the main light, the front center, and the background lights on both sides.

[0138] When a high pitch is detected, the brightness of the main light increases to 255, the maximum value, according to the weight ratio, and the color switches to red;

[0139] The brightness of the background lights on both sides is increased to 150 according to the weight, and gradually changes to pink, forming a spatial lighting effect with a prominent center and auxiliary lights on both sides, highlighting the singer's position.

[0140] Joint Vector accelerates command transmission. Joint Vector Z encodes high-frequency spectral characteristics and lighting parameters into a single signal, which is quickly transmitted to the lighting controller through a protocol control signal, avoiding the delay caused by independent transmission of multiple parameters and enabling the light to respond to high-frequency changes within 0.1 seconds.

[0141] When entering the accompaniment section, the system achieves the following effects through the joint codec module:

[0142] Maximal pooling in time domain extracts rhythm peaks:

[0143] Perform time-domain maximum pooling on the low-frequency band of the Mel spectrum to capture the peak moment of the drum rhythm and ensure that the light flashes in sync with the beat.

[0144] Spatial weighted coding to achieve dynamic light matrix:

[0145] If there is a ring light strip on the top of the box, when a 4 / 4 drum beat is detected, the ring light strip areas will be lit in sequence according to the weight, forming a clockwise flowing light effect. The brightness of each area is adjusted according to the weight ratio to enhance the spatial flow of the rhythm.

[0146] The joint vector compresses the feature dimension, encoding the drum spectrum characteristics and the spatial weight parameters of the ring light strip into a low-dimensional joint vector, reducing the amount of data. This enables the lighting controller to parse and execute synchronous changes in multiple area lights in real time, avoiding flicker delays caused by large data volumes.

[0147] The calibration module is further included, and is used to receive an actual delay T measured externally. actual , dynamically adjust the prediction step size M, the specific process is:

[0148] ;

[0149] Where T target is the preset target delay value;

[0150] Adapt to environmental delay and dynamically compensate for system errors:

[0151] By measuring the actual delay T in real time actual and with the target delay T target In contrast, the prediction step size M is automatically adjusted to enable the system to adapt to hardware differences or network fluctuations in different KTV boxes, avoiding synchronization deviations caused by fixed parameters. There is no need for technicians to manually debug delay parameters. The system automatically completes calibration, which is particularly suitable for standardized deployment of chain KTV stores and reduces operation and maintenance costs. When KTV equipment ages or is temporarily replaced, the calibration module continuously monitors delay changes and adjusts the prediction step size to ensure that the lighting synchronization effect does not decay over time.

[0152] For example, when calibrating hardware differences between different boxes:

[0153] Box A of a certain KTV chain uses an old lighting controller with high processing latency, while Box B uses a newer device with lower latency.

[0154] The initial state of box A is:

[0155] System default T target =50ms, first measurement T actual =60ms, according to the formula The prediction step size M is reduced to 5 / 6 of its original value. The LSTM model now predicts lighting commands 10ms in advance, compensating for hardware delays and ensuring that the lights and sound are synchronized during the singer's performance. For example, the light changes color without lag during high notes.

[0156] The initial state of box B is: Measure T actua =40ms, then , the prediction step size M is increased to 5 / 4 of the original value to avoid the model predicting too early and causing the light to respond in advance, such as the light flashing before the drum beat arrives.

[0157] For example, dynamic calibration after a temporary equipment replacement:

[0158] A KTV private room experienced a lighting controller failure, which led to a temporary replacement with a backup device. This caused the actual latency to increase suddenly from 50ms to 70ms.

[0159] The system monitors T in real time actua =70ms, automatic execution , shorten the prediction step and shift the prediction time point of the LSTM model back by 20ms.

[0160] Effect: The lighting lag problem originally caused by equipment replacement has been eliminated. When the user sings high-pitched passages, the lighting brightness increases in sync with the sound pressure peak, and there is no lag.

[0161] For example, when calibrating the delay fluctuation during multi-person interaction, when multiple people in a KTV room sing simultaneously using microphones or turning on the reverberation effect, the audio processing chain delay may fluctuate, for example, from 50ms to 55ms:

[0162] The calibration module continuously monitors T actual , adjust M immediately after each fluctuation. For example, when the delay increases to 55ms, M is adjusted to 50 / 55 of the original value, ensuring that the lighting responds accurately to each note.

[0163] When multiple people sing in chorus, even if the reverberation causes the sound to be delayed and changed, the lighting can still change synchronously with the moment of each singer's vocalization. For example, the light color mixing effect of the harmony part is exactly the same as the moment of sound superposition.

[0164] The process of obtaining the difference D is as follows:

[0165] ;

[0166] Where α is the difference index of the arbitration module, which is used to quantitatively predict the future lighting instructions C pred With real-time lighting instructions C real ; c∈{R,G,B} traverses the three color channels of red (Red), green (Green), and blue (Blue), covering the color dimension of the light; I c,pred is the predicted brightness value of channel c, such as an integer from 0 to 255, representing the brightness intensity of the light in this color channel; I c,real is the brightness value of channel c calculated in real time and output by the real-time processing module; β is the weight coefficient of frequency difference, which is used to balance the difference contribution of color and frequency dimensions, F freq,pred is the predicted light flicker frequency, Ffreq,real The light flashing frequency is calculated in real time.

[0167] The difference D simultaneously quantifies the differences in color channels (R / G / B brightness) and frequency dimensions, avoiding arbitration bias caused by single-dimensional judgment.

[0168] For example, when the predicted light color is close to the real-time calculated result but the flashing frequency is different, the frequency difference item triggers arbitration switching to ensure that the lighting rhythm is synchronized with the audio.

[0169] The importance of color and frequency differences can be flexibly adjusted through the weight coefficients α and β to adapt to different audio scenarios.

[0170] For example, when singing lyrical songs in KTV, the emphasis can be placed on color differences (increasing α), while when playing dance music, the emphasis can be placed on frequency differences (increasing β), thereby improving scene adaptability.

[0171] Compare the multi-dimensional differences between predicted and actual lighting effects in real time, and immediately switch commands when the difference exceeds the threshold to avoid sudden changes in lighting color or rhythm dislocation caused by prediction errors, making the lighting effects more consistent with the audio content.

[0172] In the actual KTV scene;

[0173] Scene 1: Arbitration of color differences in lyrical song performances:

[0174] When a user sings "Red Bean," the LSTM predicts that the light color corresponding to the next high note is warm red (R=255, G=100, B=50). However, the real-time processing module calculates the color to be orange (R=255, G=150, B=30) due to transient audio noise.

[0175] The color channel difference is calculated to be 53.85, and then the frequency difference is calculated. Assuming that the flicker frequencies of the two are the same, |Fpred−Freal|=0;

[0176] If α=0.8 and β=0.2, then D=0.8×53.85+0.2×0=43.08, which exceeds the threshold δ=30, and the system outputs the predicted warm red instruction.

[0177] Effect: Avoid color shift caused by noise, keep the lighting color consistent with the high-pitched emotion (warmth), and enhance the immersive feeling of the performance;

[0178] Scenario 2: Frequency difference arbitration in dance music accompaniment:

[0179] When playing the accompaniment of "Princess," the LSTM predicts that the light flashing frequency corresponding to the drum beat is 120 BPM, that is, once per beat. However, the real-time processing module calculates it as 100 BPM due to low-frequency energy fluctuations:

[0180] Perform difference calculation, color channel difference: Assuming the colors are consistent, the color channel difference is 0;

[0181] The frequency difference is |120−100|=20;

[0182] If α=0.3, β=0.7, then D=0.3×0+0.7×20=14, which exceeds the threshold δ=10, and the system outputs a predicted frequency instruction of 120BPM.

[0183] That is, the flashing of lights is strictly synchronized with the beat of the music, avoiding rhythm lag caused by low-frequency fluctuations and enhancing the dynamic atmosphere of the dance music.

[0184] When the user sang "Love is Doubt," LSTM predicted the light to be purple (R=150, G=50, B=200) with a flashing frequency of 110 BPM. The real-time calculation result was blue (R=50, G=50, B=255) with a frequency of 90 BPM:

[0185] Perform difference calculation and calculate the color channel difference to be 114.13;

[0186] The frequency difference is |110−90|=20;

[0187] If α=0.6 and β=0.4, then D=0.6×114.13+0.4×20=68.478+8=76.478, which far exceeds the threshold δ, δ=30, and the system is forced to output the predicted instruction.

[0188] At the same time, correct color and rhythm deviations to ensure that the lighting effects match the rock style of the song and avoid visual confusion.

[0189] A device for reducing audio and lighting synchronization delay, the device comprising:

[0190] Audio codec chip, used to realize audio feature extraction;

[0191] Dual-core processor for running prediction module and real-time processing module;

[0192] Training coprocessor for LSTM weight matrix generation;

[0193] Programmable logic array for generating arbitration instructions and implementing joint encoding and decoding.

[0194] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0195] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0196] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A system for reducing audio and lighting synchronization delay, characterized in that: include: Audio feature extraction module, used to convert the input audio signal into Mel spectrum feature S mel ; Lighting parameter generation module, used to generate lighting control parameters V light ; The prediction module is used to predict the Mel spectrum features S in the historical data through LSTM. mel and lighting control parameters V light Predicting future lighting instructions C pred ,LSTM contains switchable weight matrices; Real-time processing module, used to process the current Mel spectrum feature S through the FFT unit mel Generate real-time lighting instructions C real ; Arbitration module, used to compare future light instructions C pred With real-time lighting instructions C real When D>δ, the future lighting instruction C is output. pred Otherwise, output real-time lighting instruction C real As an arbitration instruction, δ is the preset threshold; The joint encoding and decoding module is used to convert the Mel spectrum feature S corresponding to the arbitration instruction mel and lighting control parameters V light Jointly encoded into a joint vector Z and decoded into a protocol control signal; The prediction module includes: The human voice energy ratio calculation unit is used to calculate the human voice energy ratio. The specific process is as follows: , where E 300-3400Hz E is the energy of the audio signal in the frequency band from 300Hz to 3400Hz, which is the main frequency range of human voice. total is the total energy of the audio signal; Weight switching unit, used in R vocal When it is greater than γ, the first weight matrix W1 is used, otherwise the second weight matrix W2 is used, and γ is a preset threshold; The LSTM training module is further included, and the LSTM training module generates a first weight matrix W1 and a second weight matrix W2 through the following steps: Step a: Collect the audio dataset Dvocal for scenes dominated by human voice and the dataset Dmusic for scenes dominated by music; Step b: Construct an LSTM network architecture, which includes an input layer, a hidden layer, an output layer, and a trainable weight matrix W; Step c: Define the loss function: ; Where W is the weight matrix of the LSTM network; is the predicted lighting instruction of the i-th sample under weight W; is the real lighting instruction of the i-th sample; N is the number of training samples; Step d: Use the gradient descent algorithm combined with the time back propagation algorithm to iteratively optimize the weight matrix: Calculate the gradient of the loss function with respect to the weights based on time backpropagation: ; Update the weight matrix: , where η is the learning rate, is the loss function Gradient of the weight matrix W; Apply the process from step a to step d to the audio dataset Dvocal for scenes dominated by human voices to optimize and obtain an optimized first weight matrix W1; Applying steps a to d to the music-dominant scene dataset Dmusic to optimize the process, and obtaining an optimized second weight matrix W2; The LSTM training module is deployed on a cloud server, and the first weight matrix W1 and the second weight matrix W2 are updated to the local prediction module via a wireless network; The process of obtaining the difference D is as follows: ； Where α is the difference index of the arbitration module; c∈{R,G,B} is the color dimension of the light that traverses the three color channels of red, green, and blue; I c,pred is the predicted brightness value of channel c; I c,real is the brightness value of channel c calculated in real time; β is the weight coefficient of frequency difference; F freq,pred is the predicted light flickering frequency; F freq,real The light flashing frequency is calculated in real time.

2. The system for reducing audio and lighting synchronization delay according to claim 1, characterized in that: The LSTM training module is also used to perform the following process: We added vocal feature enhancement to the Dvocal audio dataset for vocal-dominated scenes. This enhancement includes fundamental frequency detection and enhancement, and harmonic structure analysis and optimization. Add low-frequency enhancement to the music-dominant scene dataset Dmusic. The enhancements include beat detection and rhythm pattern extraction, and low-frequency energy envelope smoothing. Use the time backpropagation algorithm to expand the time series dependency of LSTM and calculate the gradient of the loss function with respect to the weight in step d ; Batch normalization and Dropout techniques are applied during training to prevent overfitting.

3. The system for reducing audio and lighting synchronization delay according to claim 1, characterized in that: The specific processing process of the joint encoding and decoding module is as follows: Mel spectrum feature S mel Perform time domain maximum pooling. The specific process is: , where S mel (t, f) is the original Mel spectrum feature, where t represents the time dimension, f represents the frequency dimension, and max t To obtain the maximum value in the time dimension t; Light control parameter V light Perform spatial weighted coding. The specific process is as follows: , where w i is the spatial position weight, which is pre-calculated based on the geometric relationship of the light layout. i are the control parameters of each lighting unit; Generate the joint vector Z. The specific process is: , b is the bias vector, Refers to the concatenation of the Mel spectrum feature Smel′ after the maximum pooling in the time domain and the light parameter Vlight′ after the spatial weighted encoding.

4. The system for reducing audio and lighting synchronization delay according to claim 1, characterized in that: The calibration module is further included, and the calibration module is used to receive the actual delay T actual , and dynamically adjust the prediction step size M. The specific process is: ； Where T target is the preset target delay value.

5. A device for reducing audio and lighting synchronization delay, the device being based on the system according to any one of claims 1 to 4, characterized in that: The device comprises: Audio codec chip, used to realize audio feature extraction; Dual-core processor for running prediction module and real-time processing module; Training coprocessor for LSTM weight matrix generation; Programmable logic array for generating arbitration instructions and implementing joint encoding and decoding.

Images