Light effect sound control method and device, equipment, medium and product

Abstract

The application relates to a lamp effect sound control method and device, equipment, medium and product, the method comprising: acquiring audio data generated by a pickup unit connected with a current embedding unit when recording environmental sound, and extracting an audio feature sequence of the audio data; inputting the audio feature sequence into a lamp effect mapping model running in the current embedding unit to predict a lamp effect type after extracting deep semantics, wherein the lamp effect mapping model is trained to convergence by using training samples with labeled lamp effect types in a training data set, and part of the training samples are obtained by synthesizing different audio feature sequences and the same impulse response fingerprint; and calling a lamp effect control instruction corresponding to the lamp effect type to control a lamp effect display unit connected with the current embedding unit to play a corresponding lamp effect. The application can adapt to recording conditions, accurately determine the corresponding lamp effect type according to deep semantics in environmental sound, and make the control process of the lamp effect sound control equipment more intelligent.

Description

Technical Field

[0001] This application relates to the field of lighting technology, and in particular to a method, apparatus, device, medium, and product for controlling lighting effects by sound. Background Technology

[0002] Lighting effect equipment serves multiple purposes, including displaying information and creating ambiance. Its applications are wide-ranging, and its intelligence is constantly evolving to meet diverse needs. Voice-controlled lighting effect equipment is a relatively new type. It recognizes user voice commands, parses them into corresponding lighting effect control commands, and then controls the relevant lighting effect display units to play the desired effects, greatly simplifying the use of lighting effect equipment.

[0003] Currently, voice-controlled lighting effects devices often analyze voice commands based on the frequency domain characteristics of the audio signals generated by users. For example, they determine the corresponding voice command based on the amplitude or frequency domain data of the audio signal. This method relies on the spectral characteristics of the user's speech, rather than the information content of the user's speech. Therefore, its level of intelligence is limited, resulting in relatively simple lighting effects.

[0004] On the other hand, lighting effect voice control devices are usually implemented using embedded chips. These chips have limited computing power and storage space. If the lighting effect voice control device is to understand the high-level semantics in the user's speech, a deep learning model capable of high-level semantic processing needs to be written. Deep learning models have relatively high hardware requirements. If a high-performance chip is replaced, the hardware cost will inevitably increase sharply. If an externally trained model is run under existing hardware conditions, the performance of the trained model will be reduced due to changes in recording conditions after model migration. Therefore, how to upgrade the quality of lighting effect voice control devices under the multiple dilemmas of hardware constraints and intelligent requirements is a problem that those skilled in the art need to overcome. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a lighting effect sound control method and corresponding devices, equipment, non-volatile readable storage media, and computer program products.

[0006] According to one aspect of this application, a method for controlling lighting effects by sound is provided, comprising the following steps:

[0007] Acquire audio data generated by the pickup unit connected to the current embedded unit by recording ambient sound, and extract the audio feature sequence of the audio data;

[0008] The audio feature sequence is input into the lighting effect mapping model running in the current embedding unit to extract deep semantics and predict the lighting effect type. The lighting effect mapping model is pre-trained to convergence using training samples labeled with lighting effect types in the training dataset. Some of the training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint.

[0009] The corresponding lighting effect control command for the lighting effect type is invoked to control the lighting effect display unit connected to the currently embedded unit to play the corresponding lighting effect.

[0010] In an optional embodiment, before acquiring the audio data generated by the pickup unit recording ambient sound, the following steps are included:

[0011] Prepare the training dataset such that it contains multiple sample groups, each sample group contains multiple training samples, and the multiple training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint, and the impulse response fingerprints in different sample groups are different from each other;

[0012] The lighting effect mapping model is trained by iteratively calling the training samples in the training dataset using a training device until the lighting effect mapping model is trained to a convergent state, and the quantized configuration file of the lighting effect mapping model is obtained.

[0013] The configuration file is written into the current embedded unit so that the configuration file is applied in the current embedded unit to run the lighting effect mapping model.

[0014] In an optional embodiment, the lighting effect mapping model is trained by iteratively calling the training samples in the training dataset using a training device, including:

[0015] Configure multiple instances of the lighting effect mapping model, wherein the network complexity of each instance is set to decrease sequentially.

[0016] Using the training dataset, a single instance among the plurality of instances is trained to a convergent state to obtain a converged instance;

[0017] The current instance, whose network complexity is less than that of the converged instance, is constructed as a knowledge distillation network with the previous converged instance;

[0018] Based on the knowledge distillation network, the current instance is trained to a convergent state using the training dataset;

[0019] Iterate until the instance with the minimum network complexity is trained to a convergent state, and determine it as the final usable lighting effect mapping model.

[0020] In an optional embodiment, preparing the training dataset includes:

[0021] Acquire multiple impulse response files, each of which stores an impulse response fingerprint. Each impulse response fingerprint is used to describe the impulse response characteristics of a single recording condition.

[0022] Obtain a pre-defined audio dataset, which stores multiple sample audio files pre-labeled with lighting effect types;

[0023] The audio feature sequences of each sample audio in the audio dataset are respectively matched one-to-one with the impulse response fingerprints in each impulse response file to generate multiple training samples.

[0024] The training samples are associated with their corresponding lighting effect types and stored as the training dataset.

[0025] In an optional embodiment, before acquiring multiple impulse response files, the following steps are included:

[0026] Record the ambient sound generated after playing the audio samples under multiple recording conditions to obtain multiple audio files accordingly;

[0027] The audio samples and each of the audio files are respectively transformed into frequency domain data;

[0028] Referring to the frequency domain data of the audio samples, the impulse response fingerprints of the frequency domain data of each audio file are extracted and stored as corresponding impulse response files.

[0029] In an optional embodiment, referring to the frequency domain data of the audio samples, the impulse response fingerprint of the frequency domain data of each audio file is extracted, including:

[0030] Perform a deconvolution operation on the frequency domain data of the audio sample to obtain the corresponding deconvolution feature information of the audio sample;

[0031] The dot product of the frequency domain data and the deconvolution feature information of each audio file is calculated and used as the impulse response fingerprint corresponding to that audio file.

[0032] In an optional embodiment, the audio feature sequence is input into a lighting effect mapping model running in the current embedding unit to extract deep semantics and predict the lighting effect type. The lighting effect mapping model performs the following steps:

[0033] A feature extraction network is used to extract deep semantic information from the audio feature sequence of the input.

[0034] A classifier is used to map the deep semantic information to a preset classification space to obtain the classification probability corresponding to each classification in the classification space;

[0035] The lighting effect type corresponding to the category with the highest classification probability is determined as the predicted lighting effect type.

[0036] According to another aspect of this application, a lighting effect sound control device is provided, comprising:

[0037] The audio acquisition module is configured to acquire audio data generated by the pickup unit connected to the current embedded unit recording ambient sound, and extract the audio feature sequence of the audio data;

[0038] The type prediction module is configured to input the audio feature sequence into the lighting effect mapping model running in the current embedding unit to extract deep semantics and then predict the lighting effect type. The lighting effect mapping model is pre-trained to convergence using training samples labeled with lighting effect types in the training dataset, wherein some training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint.

[0039] The lighting effect control module is configured to call the lighting effect control command corresponding to the lighting effect type to control the lighting effect display unit connected to the currently embedded unit to play the corresponding lighting effect.

[0040] According to another aspect of this application, a lighting effect sound control device is provided, including an embedded unit and a sound pickup unit and a lighting effect display unit connected to the embedded unit. The embedded unit includes a central processing unit and a memory. The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the lighting effect sound control method described in this application.

[0041] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the aforementioned light effect sound control method in the form of computer-readable instructions. When the computer program is invoked by a computer, it executes the steps included in the method.

[0042] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0043] Compared with existing technologies, this application has several technological advantages, including but not limited to:

[0044] First, this application writes a pre-trained lighting effect mapping model into the embedding unit, acquires audio data using a sound pickup device, encodes the audio data to obtain its audio feature sequence, and uses the lighting effect mapping model to infer and predict the corresponding lighting effect type based on the audio feature sequence. According to the lighting effect type, the lighting effect display unit is controlled to play the corresponding lighting effect. Since the audio feature sequence carries the original semantics of the audio data, and the lighting effect mapping model is trained to extract deep semantics based on the audio feature sequence, the corresponding lighting effect type can be predicted based on the understanding of deep semantics. Therefore, the complexity of understanding the audio data can be improved, the sound information in the audio data can be understood more accurately, and more complex sound control functions can be realized.

[0045] Secondly, the lighting effect mapping model of this application is generated by training with specific training samples. The training samples synthesize impulse response fingerprints based on the audio feature sequences of audio data to effectively simulate the recording conditions of audio data. The lighting effect mapping model prepared with the training samples has the ability to understand the audio feature sequences of audio data generated under various recording conditions. When it is transferred to the embedded unit for operation, it can adapt to the hardware conditions of the embedded unit and the pickup device connected to it. Based on the audio feature sequences of audio data obtained by the pickup device, while being compatible with the impulse response characteristics corresponding to the recording conditions of the pickup device, it can more accurately understand the deep semantics in the audio data. Based on the deep semantics, it can accurately understand the lighting effect type pointed to by the sound information in the audio data, thereby controlling the lighting effect display unit to play the corresponding lighting effect.

[0046] Furthermore, the implementation of this application does not require any changes to the hardware conditions of the lighting effect voice control device, but it can ensure that the inference capability remains basically unchanged after the lighting effect mapping model is migrated and written into the embedded unit. While maintaining the overall cost at a basically unchanged level, it can also effectively improve the human-computer interaction capability of the lighting effect voice control device in terms of voice control, taking into account the balance and optimization between hardware conditions and the degree of intelligence, and comprehensively improving the quality of the lighting effect voice control device in terms of voice control. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the principle of a lighting effect sound control device according to this application;

[0049] Figure 2This is a schematic diagram of the model architecture of a lighting effect mapping model according to this application;

[0050] Figure 3 This is a flowchart illustrating one embodiment of the lighting effect sound control method of this application;

[0051] Figure 4 This is a schematic diagram of the process for preparing a lighting effect mapping model for the embedded unit in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the training lighting effect mapping model in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram illustrating the process of constructing the training dataset in an embodiment of this application;

[0054] Figure 7 This is a schematic diagram of the process for obtaining impulse response fingerprints in an embodiment of this application;

[0055] Figure 8 Here is a spectrogram of an example audio sample. Figure 9 The two graphs are spectrograms of the audio file recorded after the audio sample was played. They correspond in terms of time and frequency. Figure 9 Compared to Figure 8 The highlighted portion shows the image corresponding to the impulse response characteristics;

[0056] Figure 10 This is a schematic diagram of the process for obtaining impulse response fingerprints in an embodiment of this application;

[0057] Figure 11 This is a flowchart illustrating the reasoning process of the lighting effect mapping model in this application embodiment;

[0058] Figure 12 This is a flowchart illustrating one embodiment of the model training method of this application;

[0059] Figure 13 This is a schematic diagram of the lighting effect sound control device of this application;

[0060] Figure 14 This is a schematic diagram of the structure of a computer device that can be used as a training device in this application. Detailed Implementation

[0061] Please see Figure 1 In the exemplary block diagram of a lighting effect sound control device provided in this application, the lighting effect sound control device includes an embedded unit, a sound pickup unit, and a lighting effect display unit, wherein:

[0062] The embedded unit can be implemented using various embedded chips, such as Bluetooth SoC, WiFi SoC, MCU, DSP, etc., which typically include a central processing unit and a memory, mainly used to store and execute program instructions to achieve corresponding functions, such as running the lighting effect mapping model of this application and other related control logic.

[0063] The pickup unit refers to a device suitable for collecting audio data of ambient sound, such as a microphone or other peripherals that carry a microphone.

[0064] The lighting effect display unit can contain single or multiple data points, used to play corresponding lighting effects according to lighting effect control commands, achieving effects such as information display and ambiance decoration. The lighting effects can be preset by those skilled in the art and pre-stored in the embedded unit and / or lighting effect display unit, and applied when needed.

[0065] When the lighting effect sound control device is running, the sound pickup unit can collect ambient sound in real time and generate corresponding audio data. Then, the embedding unit understands the sound information in the ambient sound based on the audio data, and infers and determines the corresponding lighting effect type based on the sound information. Then, it uses the lighting effect control command corresponding to the lighting effect type to control one or more of the lighting effect display units to play the corresponding lighting effect.

[0066] The sound information can be any sound information that can be parsed into instructions, including but not limited to human voice information, musical instrument sound information, etc., and its form of expression can be generated by a natural person speaking, or by a player playing music files, audio files, etc.

[0067] Please see Figure 2 In the exemplary network architecture diagram of the lighting effect control model provided in this application, the lighting effect control model includes a feature extraction network and a classifier. The feature extraction network is responsible for extracting deep semantic information from the audio feature sequence collected from the audio data to realize feature representation. The classifier is responsible for determining the classification probability corresponding to each lighting effect type in the classification space based on the deep semantic information, so as to determine the lighting effect type with the highest classification probability as the lighting effect type corresponding to the audio data.

[0068] The feature extraction network can be implemented using any deep learning model suitable for processing sequential information, including but not limited to known basic models such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTMs), and Transformer encoders. This leverages the advantages of these basic models in processing sequential information to achieve more efficient and accurate semantic understanding. Among these, LSTMs and Transformer encoders are preferred due to their stronger advantages in context understanding, especially the Transformer encoder, which can also incorporate attention mechanisms to highlight key semantics and is better suited for processing speech information. Of course, any variant versions of these basic models can also be used. Furthermore, to save on training and implementation costs, Convolutional Neural Networks (CNNs) can also be used.

[0069] The classifier can be either a binary classifier or a multi-classifier, depending on the number of preset lighting effect types corresponding to the lighting effect sound control device. When using a multi-classifier, a Softmax function can be used; when using a binary classifier, a Sigmoid function can be used. The number of categories in the classifier's classification space corresponds to the number of lighting effect types that the lighting effect sound control device can generate. For example, lighting effect types can include classical, rock, slow rock, etc., and corresponding categories are provided in the classifier's classification space.

[0070] The lighting effect control model can be trained to convergence in advance on a dedicated training device using a preset training dataset, then saved as a corresponding file and migrated and written into the embedding unit for execution.

[0071] Based on the above explanation of principles, please refer to Figure 3 According to a lighting effect sound control method provided in this application, in one embodiment, it includes the following steps:

[0072] Step S1100: Obtain audio data generated by the pickup unit connected to the current embedded unit recording ambient sound, and extract the audio feature sequence of the audio data;

[0073] In an exemplary scenario, a computer program product implementing the lighting effect sound control method according to this application is run in the embedded unit of the lighting effect sound control device. This embedded unit serves as the current embedded unit, and by running the computer program, the various steps of the method are executed.

[0074] The lighting effect sound control device provides a sound pickup unit connected to the embedded unit. The sound pickup unit is responsible for collecting ambient sound in real time and obtaining the audio data of the ambient sound.

[0075] In one embodiment, the audio data of the pickup unit refers to audio data corresponding to a frequency band perceptible to the human ear, ranging from 20Hz to 20kHz. The pickup device can sample ambient sound at a certain preset sampling frequency to obtain the aforementioned audio data.

[0076] The sampling frequency is limited by the sound card module serving the pickup device. In some embodiments, it can also be preset by those skilled in the art as needed, usually by selecting one of several levels. For example, the audio data can be formed by sampling at any one of the five levels: 11025Hz, 22050Hz, 24000Hz, 44100Hz, and 48000Hz.

[0077] For audio data from a microphone, conventional audio preprocessing operations can be used. First, pre-emphasis is applied. Then, considering a moderate frame shift, the data is segmented and windowed to obtain multiple data frames. Next, a short-time Fourier transform is performed on each data frame to convert it from time-domain data to frequency-domain data. This data is then filtered by a Mel filter to obtain the Mel spectrum. In some embodiments, this process can be implemented using the mature function FBank. Based on the Mel spectrum, Mel cepstral coefficients are further extracted. In some embodiments, the mature MFCC function can be used to obtain the Mel cepstral coefficients. The Mel cepstral coefficients provide a preliminary feature representation of the data frames. All the initial feature representations of the audio data are further constructed into an audio feature sequence, which is the encoding result of the audio data, thus achieving a preliminary feature representation of the audio data.

[0078] The process of encoding audio data to obtain its corresponding audio feature sequence as disclosed above is also applicable to other steps in this application that require determining audio feature sequences.

[0079] In one embodiment, to ensure compatibility with the lighting effect mapping model of this application, a preset effective duration can be used. Each time, audio data generated by the pickup unit is acquired according to this effective duration to determine its corresponding audio feature sequence, thus standardizing the information length of the audio feature sequence. The effective duration can be flexibly set by those skilled in the art in conjunction with the input parameter requirements of the lighting effect mapping model.

[0080] Step S1200: Input the audio feature sequence into the lighting effect mapping model running in the current embedding unit to extract deep semantics and predict the lighting effect type. The lighting effect mapping model is trained to convergence using training samples labeled with lighting effect types in the training dataset. Some of the training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint.

[0081] The audio feature sequence of the audio data from the pickup unit is further input into the lighting effect mapping model of this application. The lighting effect mapping model is pre-modeled to map the corresponding lighting effect type according to the audio feature sequence, and is pre-trained to a convergent state using the corresponding training dataset before being transferred to the embedding unit for operation.

[0082] After receiving the audio feature sequence, the lighting effect mapping model first extracts its deep semantic information. Then, it maps this deep semantic information to various lighting effect types, obtaining the classification probability corresponding to each type. The lighting effect type with the highest classification probability is determined as the type mapped to the audio feature sequence and is thus the target type. Therefore, the process by which the lighting effect mapping model processes the audio feature sequence to determine its target type essentially involves achieving a deep semantic understanding of the audio feature sequence and determining the target type based on the understanding result.

[0083] Because the lighting effect mapping model is trained using training samples from the training dataset optimized in this application, it acquires the ability to be compatible with the hardware conditions and environmental conditions of the sound pickup device. Therefore, based on the compatibility of the impulse response characteristics, it can perform accurate deep semantic understanding based on the audio feature sequence in order to accurately predict the corresponding lighting effect type.

[0084] The ability of the lighting effect mapping model to be compatible with the impulse response characteristics stems from the pre-optimization of the training dataset it uses.

[0085] In one embodiment, at least a portion of the training samples in the training dataset have the same impulse response fingerprint superimposed on the audio feature sequences corresponding to different sample audios. Therefore, after such training samples are input into the lighting effect mapping model for training, the lighting effect mapping model can include the understanding of impulse response characteristics when performing semantic understanding on the input audio feature sequences, thereby achieving accurate feature representation of the audio feature sequences.

[0086] In another embodiment, the training dataset may contain training samples from different parts, wherein the training samples from the same part are synthesized from different audio feature sequences corresponding to the sample audio based on the same impulse response fingerprint, but the impulse response fingerprints used for the training samples from different parts are different.

[0087] The impulse response fingerprint can be prepared in advance, for example, extracted from audio data generated by actual recording equipment, or it can be manually customized, and so on. Impulse response (IR) can be defined as the time-domain (time-to-amplitude) response characteristic of a system under test when a pulse excitation signal is input. The "system under test" referred to here can be as small as a microphone, a speaker unit, or a filter on an equalizer, or as large as a concert hall, a stadium, or an entire sound system, or the sum of a venue and its sound system. For example, for the pickup unit of the lighting control device in this application, the "system under test" can be understood as the hardware conditions of the pickup unit itself and the environmental conditions in which it is located. Therefore, the impulse response characteristic can be considered an acoustic "fingerprint" of a "system." The impulse response characteristic contains a wealth of information about the acoustic system, including arrival time, frequency components of the direct sound, discrete reflected sound, reverberation attenuation characteristics, signal-to-noise ratio, necessary information for analyzing speech intelligibility, and the overall frequency response.

[0088] It is easy to understand that the more impulse response fingerprints contained in the training dataset, the stronger the generalization ability of the training samples to impulse response characteristics. The resulting lighting effect mapping model, trained with this model, will have a stronger ability to accurately understand semantics under different hardware and / or environmental conditions. There are many types of lighting effect voice control devices, and their pickup units may also vary, as do their application environments. For the lighting effect mapping model embedded in the lighting effect voice control device, if it possesses this compatibility capability, it will significantly benefit the mass production of the device, helping to reduce mass production costs and achieve economies of scale.

[0089] The training dataset, by synthesizing impulse response fingerprints based on the audio feature sequences of the sample audio, helps to obtain the sample audio at a lower cost and higher efficiency. For example, various music data can be obtained from the network and used as sample audio. However, this music data may be refined products lacking relevant information on impulse response characteristics. Therefore, it is particularly necessary to synthesize impulse response fingerprints based on these sample audios. Conversely, if audio data generated by recording devices with different impulse response characteristics were obtained one by one to serve as sample audio, the implementation cost would be prohibitively high.

[0090] Step S1300: Call the lighting effect control command corresponding to the lighting effect type to control the lighting effect display unit connected to the current embedded unit to play the corresponding lighting effect.

[0091] The embedded unit typically stores pre-stored mapping data between lighting effect types and their corresponding lighting effect control commands. Thus, once the target lighting effect type is determined through the lighting effect mapping model, the lighting effect control command mapped to the target lighting effect type can be obtained from the mapping data. This lighting effect control command is then transmitted to the lighting effect display unit connected to the embedded unit, thereby controlling the lighting effect display unit to play the corresponding lighting effect.

[0092] In one embodiment, multiple lighting effect display units can cooperate to display the same lighting effect content. Accordingly, when the embedded unit transmits the lighting effect control command to the multiple lighting effect display units, each lighting effect display unit plays its respective lighting effect display process according to the lighting effect control command, and all lighting effect display units jointly display the lighting effect content.

[0093] In another embodiment, after the lighting effect mapping model transmits a lighting effect control command of a lighting effect type, such as classical, to the lighting effect display unit, the lighting effect display unit controls the lighting effect to, for example, flashing, according to a predefined protocol. Specifically, the rhythm point can be calculated based on the amplitude of the audio data to control the amplitude and duration of the flashing of the light-emitting elements in the lighting effect display unit. If the lighting effect type subsequently determined by the lighting effect mapping model does not change, there is no need to transmit the same lighting effect control command to the lighting effect display unit. Once the lighting effect mapping model subsequently generates a different lighting effect type, such as rock, it transmits the corresponding lighting effect control command, and according to a predefined protocol, controls the lighting effect display unit to scroll and display the lighting effect content according to the rhythm point of the corresponding audio data. In some embodiments, during the switching display of two different lighting effect types, a smoothing process can be performed, allowing the brightness and color values ​​of the light-emitting elements to change gradually to achieve a natural transition.

[0094] As can be seen from the above embodiments, this application has multiple technical advantages, including but not limited to:

[0095] First, this application writes a pre-trained lighting effect mapping model into the embedding unit, acquires audio data using a sound pickup device, encodes the audio data to obtain its audio feature sequence, and uses the lighting effect mapping model to infer and predict the corresponding lighting effect type based on the audio feature sequence. According to the lighting effect type, the lighting effect display unit is controlled to play the corresponding lighting effect. Since the audio feature sequence carries the original semantics of the audio data, and the lighting effect mapping model is trained to extract deep semantics based on the audio feature sequence, the corresponding lighting effect type can be predicted based on the understanding of deep semantics. Therefore, the complexity of understanding the audio data can be improved, the sound information in the audio data can be understood more accurately, and more complex sound control functions can be realized.

[0096] Secondly, the lighting effect mapping model of this application is generated by training with specific training samples. The training samples synthesize impulse response fingerprints based on the audio feature sequences of audio data to effectively simulate the recording conditions of audio data. The lighting effect mapping model prepared with the training samples has the ability to understand the audio feature sequences of audio data generated under various recording conditions. When it is transferred to the embedded unit for operation, it can adapt to the hardware conditions of the embedded unit and the pickup device connected to it. Based on the audio feature sequences of audio data obtained by the pickup device, while being compatible with the impulse response characteristics corresponding to the recording conditions of the pickup device, it can more accurately understand the deep semantics in the audio data. Based on the deep semantics, it can accurately understand the lighting effect type pointed to by the sound information in the audio data, thereby controlling the lighting effect display unit to play the corresponding lighting effect.

[0097] Furthermore, the implementation of this application does not require any changes to the hardware conditions of the lighting effect voice control device, but it can ensure that the inference capability remains basically unchanged after the lighting effect mapping model is migrated and written into the embedded unit. While maintaining the overall cost at a basically unchanged level, it can also effectively improve the human-computer interaction capability of the lighting effect voice control device in terms of voice control, taking into account the balance and optimization between hardware conditions and the degree of intelligence, and comprehensively improving the quality of the lighting effect voice control device in terms of voice control.

[0098] Based on any embodiment of this application, please refer to Figure 4 Before acquiring the audio data generated by the pickup unit recording ambient sound, the process includes:

[0099] Step S2100: Prepare the training dataset, so that the training dataset contains multiple sample groups, each sample group contains multiple training samples, and the multiple training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint, and the impulse response fingerprints in different sample groups are different from each other;

[0100] The training dataset can be pre-prepared for use. During preparation, the training dataset contains multiple sample groups, each containing a large number of training samples. These training samples are pre-labeled with their corresponding lighting effect types. Within each sample group, all training samples are derived from different audio feature sequences. These audio feature sequences are then used to synthesize the same impulse response fingerprint. In other words, all training samples within the same sample group carry the same impulse response fingerprint, thus ensuring that each impulse response fingerprint is carried by multiple training samples. Different sample groups, however, carry distinct impulse response fingerprints. Specifically, any two training samples from different sample groups are synthesized from different impulse response fingerprints. Therefore, in the training dataset, multiple impulse response fingerprints are loaded through training samples. Each impulse response fingerprint is carried by multiple training samples. Typically, different impulse response fingerprints can be matched with the audio feature sequences of the same batch of audio samples to prepare training samples in batches. Thus, the number of samples can be multiplied according to the number of impulse response fingerprints. Furthermore, the same audio feature sequence can also be synthesized with different impulse response fingerprints, so that multiple audio feature sequences of multiple audio samples and multiple impulse response fingerprints can fully interact and combine, and the obtained training dataset can fully generalize various features.

[0101] Step S2200: Use the training device to iteratively call the training samples in the training dataset to train the lighting effect mapping model, train the lighting effect mapping model to a convergent state, and obtain the quantized configuration file of the lighting effect mapping model.

[0102] After obtaining the training dataset, the lighting effect mapping model of this application can be iteratively trained using the various training samples on a training device such as a computer until it reaches a convergent state. Since the training samples are based on the audio feature sequences of the sample audio, data augmentation is achieved through synthetic impulse response fingerprints, ensuring that a large number of training samples contain diverse features. Therefore, when the lighting effect mapping model is trained to a convergent state using the training dataset, it possesses the ability to accurately understand diverse features. Because impulse response fingerprints are used for data augmentation, the lighting effect mapping model can adapt to various hardware and / or environmental conditions, enabling accurate semantic understanding of the audio feature sequences of audio data obtained under different conditions.

[0103] In some embodiments, during the process of calling training samples in the training dataset, other data augmentation techniques may be further employed to randomly discard some features from the audio feature sequences in the training samples in order to further generalize the features.

[0104] In one embodiment, for a lighting effect mapping model that has been trained to convergence, a model quantization operation is performed. The lighting effect mapping model is quantized using the IN16 or IN8 algorithm, converting the network parameters obtained from training the lighting effect mapping model from high precision to low precision, for example, reducing 32 bits to 16 bits or 8 bits, thereby reducing the storage and running space occupied by the lighting effect mapping model, making it more compatible with embedded units such as embedded chips.

[0105] After the above processing is completed to finalize the lighting effect mapping model, the lighting effect mapping model can be output as a configuration file, for example, stored as a C language file (*.c and *.h).

[0106] Step S2300: Write the configuration file into the current embedded unit to apply the configuration file and run the lighting effect mapping model in the current embedded unit.

[0107] To adapt to the processing of the embedded units in the production stage of lighting effect sound control equipment, the configuration file can be written in batches into the embedded units of each device. This enables the migration and writing of the lighting effect mapping model into the embedded unit. After the embedded unit is powered on and enabled, the lighting effect mapping model can be run to provide the corresponding mapping service.

[0108] As can be seen from the above embodiments, to meet the requirement of deploying the lighting effect mapping model of this application in the embedding unit, after training the lighting effect mapping model to convergence using training samples that incorporate impulse response fingerprints in the training dataset, this application also uses model lightweighting to compress the model scale, making it more suitable for running in the embedding unit. Based on the comprehensive advantages of lightweight characteristics and compatibility with different impulse response characteristics, it makes it efficient and feasible to perform deep semantic understanding of audio data in the embedding unit. Through actual testing, it can maximize the maintenance of the performance of the lighting effect mapping model in the training device, maintain the excellent performance of the lighting effect mapping model in the training device in lighting effect voice control devices under various hardware conditions and / or environmental conditions, and ensure a more intelligent voice control effect.

[0109] Based on any embodiment of this application, please refer to Figure 5 The lighting effect mapping model is trained by iteratively calling the training samples in the training dataset using a training device, including:

[0110] Step S2210: Configure and generate multiple instances of the lighting effect mapping model, wherein the network complexity of each instance is set to decrease sequentially;

[0111] When training a lighting effect mapping model using a training device, the computing power provided by the training device can be leveraged to iteratively train multiple instances, gradually reducing the network complexity of the lighting effect mapping model to minimize the network size of the final lighting effect mapping model, thereby further reducing the storage and running space occupied by the lighting effect mapping model.

[0112] Therefore, multiple instances of the lighting effect mapping model can be configured with different network complexities, such that the network complexity of each instance decreases from high to low. For example, the first instance may contain 8 convolutional layers, the second instance may contain 4 convolutional layers, the third instance may contain 2 convolutional layers, and so on. As long as each instance is constructed based on the same lighting effect mapping model and has a different network complexity, it is acceptable.

[0113] Step S2220: Using the training dataset, train a single instance among the multiple instances to a converged state to obtain a converged instance;

[0114] After obtaining the multiple instances, the instance with the highest network complexity, such as the first instance, is typically trained to convergence using training samples from the training dataset. Instances that have been trained to convergence can be named converged instances.

[0115] Step S2230: Construct a knowledge distillation network by combining the current instance with the previous converged instance whose network complexity is less than that of the converged instance;

[0116] Furthermore, the converged instance trained in the previous step is used as the teacher model, and an instance with relatively low network complexity, such as the second instance, is used as the student model. Following the idea of ​​knowledge distillation training, the teacher model and the student model are constructed into a knowledge distillation network. The principle of the knowledge distillation network is to use the reasoning ability already learned by the teacher model to guide the training process of the student model, thereby improving the training speed and reasoning ability of the student model. Moreover, since the student model has lower network complexity than the teacher model, it can ensure that model compression is gradually achieved.

[0117] Step S2240: Based on the knowledge distillation network, train the current instance to a convergent state using the training dataset;

[0118] Based on the knowledge distillation network, the training dataset is still used, and the teacher model and student model are synchronously input. Under the supervision of the teacher model, the student model is trained to a convergent state, and thus the student model becomes a converged instance.

[0119] Step S2250: Iterate until the instance with the minimum network complexity is trained to a convergent state, and determine it as the final usable lighting effect mapping model.

[0120] To achieve model compression, steps S2230 and S2240 can be iterated. Each time, the previous converged instance is used as the teacher model and another instance with lower network complexity is used as the student model to construct a knowledge distillation network. The instance with lower network complexity is trained, and so on, until the instance with the minimum network complexity is trained to convergence. This instance is then determined as the final usable light effect mapping model.

[0121] As mentioned above, in other embodiments, model quantization can also be performed on the instance with minimum network complexity to make the final lighting effect mapping model more miniaturized.

[0122] As can be seen from the above embodiments, the lighting effect mapping model of this application can be combined with the knowledge distillation idea to perform multi-instance compression training in order to obtain the most concise model instance, thereby compressing the model data scale and making it more suitable for deployment in embedded devices to ensure high-quality running performance.

[0123] Based on any embodiment of this application, please refer to Figure 6 Prepare the training dataset, including:

[0124] Step S2110: Obtain multiple impulse response files, each impulse response file stores an impulse response fingerprint, and each impulse response fingerprint is used to describe the impulse response characteristics of a single recording condition;

[0125] When preparing the training dataset, multiple impulse response files can be obtained beforehand for synthesizing the sample audio. Each impulse response file can store a single impulse response fingerprint, and each impulse response fingerprint, as mentioned earlier, is primarily used to describe the impulse response characteristics of a single corresponding recording condition. Similarly, the recording conditions, as previously described, can correspond to specific hardware and / or environmental conditions. For example, the impulse response characteristics generated from audio data obtained using a specific recording device in a specific environment can be used to determine the impulse response fingerprint corresponding to the recording conditions under that specific condition.

[0126] Step S2120: Obtain a preset audio dataset, which stores multiple sample audios pre-labeled with lighting effect types;

[0127] To prepare training samples, an audio dataset can be obtained, which can be pre-collected from various audio data from public networks as sample audio, and these sample audios can be manually labeled with their corresponding lighting effect types, so that a mapping relationship is established between sample audio and lighting effect types.

[0128] The sample audio can be human voice, music, etc., selected according to the actual sound content required by the lighting effect mapping model. For example, when the lighting effect mapping model needs to understand instructions given by human voice, sample audio of corresponding human voice content can be used to prepare training samples; when the lighting effect mapping model needs to understand instructions corresponding to song melody, sample audio of corresponding music content can be used to prepare training samples.

[0129] Typically, audio samples obtained from publicly available sources are mostly filtered audio, which has a weaker impulse response characteristic corresponding to the recording conditions. Therefore, further processing is required to adapt them to the needs of this application.

[0130] Step S2130: Combine the audio feature sequences of each sample audio in the audio dataset with the impulse response fingerprints in each impulse response file to generate multiple training samples.

[0131] In one embodiment, for each sample audio in the audio dataset, the conventional audio signal preprocessing method described above can be applied to determine its corresponding audio feature sequence. Then, this audio feature sequence is synthesized one-to-one with the impulse response fingerprints in each of the impulse response files. The synthesized product can then be used as the corresponding training sample. Each sample audio is synthesized one-to-one with each of the impulse response fingerprints in this process. Therefore, based on the size of the sample audio in the audio dataset, a large number of training samples can be multiplied according to the number of impulse response fingerprints. For example, if there are N impulse response fingerprints and M sample audios, then M*N training samples can be obtained. Of course, in practice, some training samples can be discarded from synthesis with individual impulse response fingerprints, or some impulse response fingerprints can be discarded from synthesis with individual sample audios, which can also achieve a certain degree of data augmentation. Those skilled in the art can implement this flexibly.

[0132] After synthesizing a massive amount of training samples in the above manner, it can be seen that training samples of the same type carrying the same impulse response fingerprint can constitute the same sample group. Therefore, according to the impulse response fingerprint, the training data group will include multiple sample groups, and each sample group contains multiple training samples. Different training samples in the same sample group are all synthesized based on the same impulse response fingerprint and the corresponding audio feature sequence of the sample audio. However, the impulse response fingerprints carried by the training samples in different sample groups are different from each other.

[0133] Step S2140: Associate the multiple training samples with their corresponding lighting effect types and store them as the training dataset.

[0134] At this point, each training sample can be further labeled according to the lighting effect type of the sample audio from which it originates, forming a mapping relationship data between training samples and lighting effect types. Then, this mapping relationship data can be stored in the training dataset.

[0135] As can be seen from the above embodiments, the training samples in the training dataset of this application can be prepared by superimposing different impulse response fingerprints on sample audio obtained through public means. The cost of obtaining them is low, and the process of automatically synthesizing impulse response fingerprints is also very efficient. At the same time, by combining the audio feature sequences of a large number of sample audios with different impulse response fingerprints to construct training samples, the semantic representation of the impulse response characteristics under different recording conditions can be implicitly contained in the massive training samples. When these are used to train the lighting effect mapping model, the lighting effect mapping model can learn the ability to understand semantics in a way that is compatible with different impulse response characteristics.

[0136] Based on any embodiment of this application, please refer to Figure 7 Before acquiring multiple impulse response files, the following steps are required:

[0137] Step S3100: Record the ambient sound generated after the audio sample is played under multiple recording conditions, and obtain multiple audio files accordingly;

[0138] Multiple different recording conditions can be pre-created, such as using different rooms, different pickup units and their combinations, to determine different recording conditions. Under these different recording conditions, the same audio sample is played, and the ambient sound generated after the playback of these audio samples is recorded, obtaining the corresponding audio data, which is then stored as a corresponding audio file. From a different perspective, each recording condition theoretically corresponds to a certain impulse response characteristic, thus enabling the acquisition of a corresponding impulse response fingerprint.

[0139] Based on the above embodiments, see Figure 8 and Figure 9 , Figure 8 This is a spectrogram of an example audio sample. Figure 9 It is based on Figure 8 The spectrum of the audio file obtained from the audio sample is shown. The horizontal axis represents time, and the vertical axis represents frequency. Highlighted areas represent frequency changes. (Comparison) Figure 8 and Figure 9 It can be seen that, Figure 8 The frequency spectrum of the mid-range sample increases over time. At each point in time, it represents a relatively singular frequency range. Figure 9 Zhongyu Figure 8 When comparing data at a unified time point, based on the same frequency point, Figure 9 It contains multiple spectral responses. Therefore, it is possible to extract the corresponding impulse response fingerprint from the frequency domain data of an audio file.

[0140] Step S3200: Convert the audio samples and each of the audio files into frequency domain data respectively;

[0141] The audio data stored in each audio file is time-domain data. In order to facilitate the extraction of impulse response fingerprints, the conventional audio signal preprocessing method described above can be used first. This is done by transforming the audio samples and each audio file into frequency-domain data. Specifically, based on obtaining each data frame through windowing operations, the corresponding frequency-domain data can be obtained by applying the short-time Fourier transform function.

[0142] Step S3300: Referring to the frequency domain data of the audio sample, extract the impulse response fingerprint of the frequency domain data of each audio file and store it as the corresponding impulse response file.

[0143] It is easy to understand that the frequency domain data of the audio file implicitly contains the impulse response characteristics under the corresponding recording conditions. Based on this, by referring to the frequency domain data of the audio sample, the impulse response fingerprint corresponding to the impulse response characteristics under the corresponding recording conditions can be extracted from the audio file by performing calculations on the frequency domain data of each audio file and the frequency domain data of the audio sample.

[0144] Finally, the impulse response fingerprints corresponding to each audio file are stored as impulse response files for easy retrieval.

[0145] As can be seen from the above embodiments, the preparation process of impulse response fingerprints is relatively convenient and efficient. Each impulse response file can be quickly obtained according to the above embodiments for synthesizing impulse response fingerprints for each sample audio. Compared with the method of preparing training samples by recordings individually, it is more efficient and faster.

[0146] Based on any embodiment of this application, please refer to Figure 10 Referring to the frequency domain data of the audio samples, the impulse response fingerprint of the frequency domain data of each audio file is extracted, including:

[0147] Step S3310: Perform a deconvolution operation on the frequency domain data of the audio sample to obtain the deconvolution feature information corresponding to the audio sample;

[0148] The impulse response and frequency domain transfer function of a system are actually the results of their forward and inverse Fourier transforms. Therefore, the following formula exists:

[0149]

[0150] Wherein, the ifft function is the short-time Fourier transform function, the fft function is the inverse fast Fourier transform function, a is the frequency domain data of the audio sample, b is the frequency domain data of the audio file, and h is the impulse response fingerprint.

[0151] Based on the transformation of the above formula, it can be determined that each impulse response fingerprint can be modeled and extracted according to the principle revealed by the following formula:

[0152] Impulse response fingerprint = Frequency domain data of audio sample * Deconvolution feature information of audio file

[0153] Based on the above principle, when obtaining the impulse response fingerprint corresponding to each audio file, the frequency domain data of each audio sample is first deconvolutionally processed to obtain the deconvolution feature information corresponding to each audio sample.

[0154] Step S3320: Calculate the dot product of the frequency domain data and the deconvolution feature information of each audio file, and use it as the impulse response fingerprint corresponding to the audio file.

[0155] Furthermore, for each audio file, the dot product of the frequency domain data of the audio file and the deconvolution feature information of the audio sample is calculated through a linear layer. This dot product result is the corresponding impulse response fingerprint of the audio file, which in turn represents the impulse response characteristics of the recording conditions that generated the audio file.

[0156] As can be seen from the above embodiments, based on mathematical methods and referring to the frequency domain data of audio samples, the corresponding impulse response fingerprints can be quickly extracted from the frequency domain data of each audio file to construct the corresponding impulse response files. The computational load is low and the extraction is rapid, which can ensure that the preparation process of the training dataset is fast, efficient and economical.

[0157] Based on any embodiment of this application, please refer to Figure 11 The audio feature sequence is input into the lighting effect mapping model running in the current embedding unit to extract deep semantics and predict the lighting effect type. The lighting effect mapping model performs the following steps:

[0158] Step S1210: Use a feature extraction network to extract deep semantic information from the audio feature sequence in the input;

[0159] Please combine Figure 2 The model architecture shown allows for the following process: when the audio feature sequence of the audio data from the pickup unit of the lighting effect sound control device is input into the lighting effect mapping model running in the current embedding unit, the feature extraction network in the lighting effect mapping model extracts the semantic features of the audio feature sequence, thereby extracting the hidden deep semantic information and realizing the representation of the deep semantic features of the audio feature sequence.

[0160] Step S1220: Use a classifier to map the deep semantic information to a preset classification space to obtain the classification probability corresponding to each classification in the classification space;

[0161] The deep semantic information is further input into the classifier in the lighting effect mapping model. In the classifier, the deep semantic information is mapped to a preset classification space through a fully connected layer. The classification space corresponds to the total number of all lighting effect types and sets the corresponding classification for each lighting effect type. Thus, the probability of mapping the deep semantic information to the corresponding classification for each lighting effect type is obtained.

[0162] Step S1230: Determine the lighting effect type corresponding to the category with the highest classification probability as the predicted lighting effect type.

[0163] It is easy to understand that the category with the highest classification probability in the classification space corresponds to the lighting effect type predicted by the audio feature sequence of the audio data obtained by the pickup unit. This enables the corresponding lighting effect type to be parsed based on the deep semantics of the audio data to trigger the corresponding lighting effect control command.

[0164] As can be seen from the above embodiments, the lighting effect mapping model of this application has a relatively lightweight network architecture, which is suitable for deployment in embedded units such as embedded chips, and achieves a more efficient and economical operating effect.

[0165] Please see Figure 12 This application also provides a model training method corresponding to the lighting effect sound control method of this application, which includes the following steps:

[0166] Step S5100: Prepare a training dataset, such that the training dataset contains multiple sample groups, each sample group contains multiple training samples, and the multiple training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint, and the impulse response fingerprints in different sample groups are different from each other;

[0167] Step S5200: Use the training device to iteratively call the training samples in the training dataset to train the lighting effect mapping model, train the lighting effect mapping model to the convergence state, and obtain the quantized configuration file of the lighting effect mapping model.

[0168] Step S5300: Write the configuration file into the embedded unit of the lighting effect sound control device, so as to apply the configuration file to the embedded unit and run the lighting effect mapping model.

[0169] Based on any embodiment of this application, the lighting effect mapping model is trained by iteratively calling the training samples in the training dataset using a training device, including:

[0170] Configure multiple instances of the lighting effect mapping model, wherein the network complexity of each instance is set to decrease sequentially.

[0171] Using the training dataset, a single instance among the plurality of instances is trained to a convergent state to obtain a converged instance;

[0172] The current instance, whose network complexity is less than that of the converged instance, is constructed as a knowledge distillation network with the previous converged instance;

[0173] Based on the knowledge distillation network, the current instance is trained to a convergent state using the training dataset;

[0174] Iterate until the instance with the minimum network complexity is trained to a convergent state, and determine it as the final usable lighting effect mapping model.

[0175] Based on any embodiment of this application, the training dataset is prepared, including:

[0176] Acquire multiple impulse response files, each of which stores an impulse response fingerprint. Each impulse response fingerprint is used to describe the impulse response characteristics of a single recording condition.

[0177] Obtain a pre-defined audio dataset, which stores multiple sample audio files pre-labeled with lighting effect types;

[0178] The audio feature sequences of each sample audio in the audio dataset are respectively matched one-to-one with the impulse response fingerprints in each impulse response file to generate multiple training samples.

[0179] The training samples are associated with their corresponding lighting effect types and stored as the training dataset.

[0180] Prior to obtaining multiple impulse response files, based on any embodiment of this application, the following steps are included:

[0181] Record the ambient sound generated after playing the audio samples under multiple recording conditions to obtain multiple audio files accordingly;

[0182] The audio samples and each of the audio files are respectively transformed into frequency domain data;

[0183] Referring to the frequency domain data of the audio samples, the impulse response fingerprints of the frequency domain data of each audio file are extracted and stored as corresponding impulse response files.

[0184] Based on any embodiment of this application, referring to the frequency domain data of the audio samples, the impulse response fingerprint of the frequency domain data of each audio file is extracted, including:

[0185] Perform a deconvolution operation on the frequency domain data of the audio sample to obtain the corresponding deconvolution feature information of the audio sample;

[0186] The dot product of the frequency domain data and the deconvolution feature information of each audio file is calculated and used as the impulse response fingerprint corresponding to that audio file.

[0187] Please see Figure 13 According to one aspect of this application, a lighting effect sound control device includes an audio acquisition module 1100, a type prediction module 1200, and a lighting effect control module 1300. The audio acquisition module 1100 is configured to acquire audio data generated by a pickup unit connected to a current embedded unit recording ambient sound, and extract the audio feature sequence of the audio data. The type prediction module 1200 is configured to input the audio feature sequence into a lighting effect mapping model running in the current embedded unit to extract deep semantics and predict the lighting effect type. The lighting effect mapping model is pre-trained to convergence using training samples labeled with lighting effect types in a training dataset, wherein some training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint. The lighting effect control module 1300 is configured to call the lighting effect control command corresponding to the lighting effect type to control the lighting effect display unit connected to the current embedded unit to play the corresponding lighting effect.

[0188] Based on any embodiment of this application, prior to the operation of the audio acquisition module 1100, the following is included: a data preparation module, configured to prepare the training dataset, such that the training dataset contains multiple sample groups, each sample group contains multiple training samples, the multiple training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint, and the impulse response fingerprints in different sample groups are different from each other; a training implementation module, configured to use a training device to iteratively call the training samples in the training dataset to train the lighting effect mapping model, train the lighting effect mapping model to a convergent state, and obtain the quantized configuration file of the lighting effect mapping model; and a model transfer module, configured to write the configuration file into the current embedding unit, so that the configuration file is applied in the current embedding unit to run the lighting effect mapping model.

[0189] Based on any embodiment of this application, the training implementation module includes: an instance configuration unit, configured to configure and generate multiple instances of the lighting effect mapping model, wherein the network complexity of each instance is set to decrease sequentially; a basic training unit, configured to train a single instance among the multiple instances to a converged state using the training dataset to obtain a converged instance; a network construction unit, configured to build a knowledge distillation network by combining the current instance with a network complexity less than that of the converged instance and the previous converged instance; a distillation training unit, configured to train the current instance to a converged state using the training dataset based on the knowledge distillation network; and an iterative optimization unit, configured to iterate until the instance with the smallest network complexity is trained to a converged state, and determine it as the final usable lighting effect mapping model.

[0190] Based on any embodiment of this application, the data preparation module includes: a fingerprint retrieval unit configured to acquire multiple impulse response files, each impulse response file storing an impulse response fingerprint, and each impulse response fingerprint describing the impulse response characteristics of a single recording condition; an audio retrieval unit configured to acquire a preset audio dataset, which stores multiple sample audios pre-labeled with lighting effect types; a synthesis processing unit configured to synthesize the audio feature sequences of each sample audio in the audio dataset with the impulse response fingerprints in each impulse response file, generating multiple training samples; and an association processing unit configured to associate the multiple training samples with their corresponding lighting effect types and store them as the training dataset.

[0191] Based on any embodiment of this application, prior to the operation of the fingerprint retrieval unit, the following is included: a sample recording unit, configured to record ambient sounds formed after the playback of audio samples under multiple recording conditions, thereby obtaining multiple audio files; a time-frequency conversion unit, configured to convert the audio samples and each of the audio files into frequency domain data respectively; and a fingerprint extraction unit, configured to extract the impulse response fingerprint of the frequency domain data of each of the audio files by referring to the frequency domain data of the audio samples, and store it as a corresponding impulse response file.

[0192] Based on any embodiment of this application, the fingerprint extraction unit includes: a convolution operation subunit, configured to perform a deconvolution operation on the frequency domain data of the audio sample to obtain the deconvolution feature information corresponding to the audio sample; and a computation extraction subunit, configured to calculate the dot product of the frequency domain data of each audio file and the deconvolution feature information as the impulse response fingerprint corresponding to the audio file.

[0193] Based on any embodiment of this application, the lighting effect mapping model in the type prediction module 1200 includes: a feature extraction unit, configured to extract deep semantic information from the audio feature sequence input by using a feature extraction network; a classification mapping unit, configured to map the deep semantic information to a preset classification space by using a classifier to obtain the classification probability corresponding to each classification in the classification space; and a type determination unit, configured to determine the lighting effect type corresponding to the classification with the highest classification probability as the predicted lighting effect type.

[0194] Another embodiment of this application provides a training device implemented using computer equipment. For example... Figure 14 The diagram shows the internal structure of the computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database may store information sequences. When the computer-readable instructions are executed by the processor, the processor can implement the model training method of this application.

[0195] The processor of this computer device provides computational and control capabilities to support the operation of the entire computer device. The memory of this computer device can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the model training method of this application. The network interface of this computer device is used for communication with a terminal.

[0196] Those skilled in the art will understand that Figure 14The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0197] In this embodiment, the processor is used to execute the specific functions of each module corresponding to each step of the lighting effect sound control method or model training method according to this application. The memory stores the program code and various types of data required to execute the above modules or sub-modules. The non-volatile readable storage medium in this embodiment stores the program code and data of the computer program product implemented according to the lighting effect sound control method or model training method of this application.

[0198] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the lighting effect sound control method or model training method of any embodiment of this application.

[0199] This application also provides a computer program product, including a computer program / instructions, which, when executed by one or more processors, implement the steps of the lighting effect sound control method or model training method described in any embodiment of this application.

[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0201] In summary, this application can adapt to the recording conditions of lighting effect sound control equipment, and determine the corresponding lighting effect type by accurately understanding the sound information based on the deep semantics in the ambient sound, making the lighting effect control process more intelligent.

Claims

1. A method for controlling lighting effects with sound, characterized in that, include: Acquire audio data generated by the pickup unit connected to the current embedded unit by recording ambient sound, and extract the audio feature sequence of the audio data; The audio feature sequence is input into the lighting effect mapping model running in the current embedding unit to extract deep semantics and predict the lighting effect type. The lighting effect mapping model is pre-trained to convergence using training samples labeled with lighting effect types in the training dataset. Some of the training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint. The impulse response fingerprint is the time-domain response characteristic of the system under test when a impulse excitation signal is input, which is composed of the hardware conditions of the pickup unit itself and the environmental conditions. The corresponding lighting effect control command for the lighting effect type is invoked to control the lighting effect display unit connected to the currently embedded unit to play the corresponding lighting effect.

2. The lighting effect sound control method according to claim 1, characterized in that, Before acquiring the audio data generated by the pickup unit recording ambient sound, the process includes: Prepare the training dataset such that it contains multiple sample groups, each sample group contains multiple training samples, and the multiple training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint, and the impulse response fingerprints in different sample groups are different from each other; The lighting effect mapping model is trained by iteratively calling the training samples in the training dataset using a training device until the lighting effect mapping model is trained to a convergent state, and the quantized configuration file of the lighting effect mapping model is obtained. The configuration file is written into the current embedded unit so that the configuration file is applied in the current embedded unit to run the lighting effect mapping model.

3. The lighting effect sound control method according to claim 2, characterized in that, The lighting effect mapping model is trained by iteratively calling the training samples in the training dataset using a training device, including: Configure multiple instances of the lighting effect mapping model, wherein the network complexity of each instance is set to decrease sequentially. Using the training dataset, a single instance among the plurality of instances is trained to a convergent state to obtain a converged instance; The current instance, whose network complexity is less than that of the converged instance, is constructed as a knowledge distillation network with the previous converged instance; Based on the knowledge distillation network, the current instance is trained to a convergent state using the training dataset; Iterate until the instance with the minimum network complexity is trained to a convergent state, and determine it as the final usable lighting effect mapping model.

4. The lighting effect sound control method according to claim 2, characterized in that, Prepare the training dataset, including: Acquire multiple impulse response files, each of which stores an impulse response fingerprint. Each impulse response fingerprint is used to describe the impulse response characteristics of a single recording condition. Obtain a pre-defined audio dataset, which stores multiple sample audio files pre-labeled with lighting effect types; The audio feature sequences of each sample audio in the audio dataset are respectively matched one-to-one with the impulse response fingerprints in each impulse response file to generate multiple training samples. The training samples are associated with their corresponding lighting effect types and stored as the training dataset.

5. The lighting effect sound control method according to claim 4, characterized in that, Before acquiring multiple impulse response files, the following steps are required: Record the ambient sound generated after playing the audio samples under multiple recording conditions to obtain multiple audio files accordingly; The audio samples and each of the audio files are respectively transformed into frequency domain data; Referring to the frequency domain data of the audio samples, the impulse response fingerprints of the frequency domain data of each audio file are extracted and stored as corresponding impulse response files.

6. The lighting effect sound control method according to claim 5, characterized in that, Referring to the frequency domain data of the audio samples, the impulse response fingerprint of the frequency domain data of each audio file is extracted, including: Perform a deconvolution operation on the frequency domain data of the audio sample to obtain the corresponding deconvolution feature information of the audio sample; The dot product of the frequency domain data and the deconvolution feature information of each audio file is calculated and used as the impulse response fingerprint corresponding to that audio file.

7. The lighting effect sound control method according to any one of claims 1 to 6, characterized in that, The audio feature sequence is input into the lighting effect mapping model running in the current embedding unit to extract deep semantics and predict the lighting effect type. The lighting effect mapping model performs the following steps: A feature extraction network is used to extract deep semantic information from the audio feature sequence of the input. A classifier is used to map the deep semantic information to a preset classification space to obtain the classification probability corresponding to each classification in the classification space; The lighting effect type corresponding to the category with the highest classification probability is determined as the predicted lighting effect type.

8. A lighting effect sound control device, characterized in that, include: The audio acquisition module is configured to acquire audio data generated by the pickup unit connected to the current embedded unit recording ambient sound, and extract the audio feature sequence of the audio data; The type prediction module is configured to input the audio feature sequence into the lighting effect mapping model running in the current embedding unit to extract deep semantics and then predict the lighting effect type. The lighting effect mapping model is pre-trained to convergence using training samples labeled with lighting effect types in the training dataset. Some of the training samples are synthesized based on different audio feature sequences and the same impulse response fingerprint. The impulse response fingerprint is the time-domain response characteristic of the system under test when a impulse excitation signal is input, which is composed of the hardware conditions of the pickup unit itself and the environmental conditions. The lighting effect control module is configured to call the lighting effect control command corresponding to the lighting effect type to control the lighting effect display unit connected to the currently embedded unit to play the corresponding lighting effect.

9. A sound-controlled lighting effect device, comprising an embedded unit and a sound pickup unit and a lighting effect display unit connected to the embedded unit, wherein the embedded unit includes a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images