Adaptive control method and system for sound equipment

Abstract

The invention belongs to the technical field of sound equipment control, and particularly relates to a sound equipment adaptive control method and system, and the method comprises the steps: a mixed sound signal collection subsystem collects an effective sound signal and a noise signal generated by an external environment, and forms a to-be-processed mixed sound signal; the noise self-adaptive filtering subsystem is internally provided with a sound abnormal noise self-adaptive filtering method, so that identification of noise signal types and key parameters and self-adaptive separation and filtering of noise signals can be completed; a visual semantic double-recognition-based adaptive control method is built in the visual semantic double-recognition-based adaptive control subsystem, image and voice double-operation authority authentication is supported, and two control modes of intelligent voice and gestures are provided; the sound equipment state detection and operation terminal is located on a computer or server equipment, can carry out real-time monitoring and manual intervention on the sound equipment state, and provides computing power and data processing services at the same time. The adaptive control efficiency of the sound equipment can be effectively improved, and the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of sound control, and in particular relates to a sound adaptive control method and system. Background Art

[0002] With the rapid development of science and technology, audio equipment plays an increasingly important role in people's daily lives. From simple audio playback devices to complex audio processing systems, the application scope of audio technology is becoming increasingly wide, covering entertainment, education, transportation, security and other fields. However, traditional audio control methods often rely on fixed parameter settings and cannot be adaptively adjusted according to real-time changes in the environment, which to some extent limits the effect of the audio system and user experience.

[0003] First, in the field of entertainment, audio playback devices such as mobile phones, computers, and smart speakers have become indispensable leisure tools in people's daily lives. However, when people are in different environments (such as at home, outdoors, in transportation, etc.) or play different types of music (such as classical, rock, jazz, etc.), their demands for sound effects are also different. Traditional audio control methods cannot automatically adjust sound effect parameters according to these changes, resulting in a significant reduction in user experience.

[0004] Secondly, in the field of transportation, as part of the vehicle's active safety system, the sound control effect of the vehicle buzzer directly affects road safety. Traditional buzzer sound control algorithms are usually based on fixed frequency and volume parameters, and cannot be adaptively adjusted according to the vehicle's environment (such as noise level, vehicle speed, vehicle position, etc.) and emergency situations. This reduces the sound reminder effect of the buzzer to a certain extent and increases the risk of traffic accidents.

[0005] In addition, with the development of emerging technologies such as smart home, autonomous driving, and wearable devices, the demand for adaptive audio control is becoming increasingly urgent. These emerging technologies require the audio system to be able to dynamically adjust according to environmental changes and user needs to achieve a more intelligent and personalized audio experience.

[0006] The research on acoustic adaptive control method has far-reaching significance, which is mainly reflected in the following aspects: 1. Improve user experience: The adaptive control method can automatically adjust audio parameters such as volume, sound quality, and sound effects according to changes in the user's environment, thereby providing a more comfortable listening experience. For example, in a noisy outdoor environment, the audio system can automatically increase the volume and optimize the sound effects to ensure that the user can clearly hear the music or sound prompts; in a quiet indoor environment, it can lower the volume and adjust the sound quality to avoid disturbing others.

[0007] 2. Enhanced security: In the field of transportation, the application of adaptive control methods can significantly improve the sound reminder effect of vehicle buzzers. By real-time monitoring of the sound environment and vehicle status around the vehicle, the sound system can automatically adjust the volume and frequency of the buzzer to better attract the attention of other drivers and pedestrians, thereby reducing the risk of traffic accidents. In addition, the adaptive control method can also be integrated with other active safety systems of the vehicle to jointly improve the overall safety of the vehicle.

[0008] 3. Promote technological progress: The research and application of audio intelligent recognition and adaptive control technology is one of the important areas of scientific and technological development. By continuously exploring and optimizing these technologies, we can promote the development of audio equipment and systems in a more intelligent and personalized direction. At the same time, the progress of these technologies will also provide strong support for technological innovation in other fields, such as smart homes, autonomous driving, wearable devices, etc.

[0009] 4. Expand application areas: The adaptive control method is not only applicable to fields such as audio playback equipment and vehicle buzzers, but can also be widely used in other scenarios that require audio control. For example, in the public broadcasting system in public places, the adaptive control method can automatically adjust the volume and sound quality according to the density of people and noise levels in different areas; in the conference system, the sound effect can be automatically adjusted according to the size and layout of the venue to ensure that every participant can get a clear listening experience.

[0010] 5. Promote sustainable development: The application of adaptive control methods helps to reduce energy waste and environmental pollution. By automatically adjusting the sound parameters to meet environmental requirements, unnecessary energy consumption and noise pollution can be reduced. In addition, with the continuous advancement of technology and the continuous expansion of application areas, adaptive control methods will also play a more active role in energy conservation, emission reduction and environmental protection.

[0011] In summary, the research on adaptive sound control methods has important practical significance and broad development prospects. By continuously exploring and optimizing these technologies, we can provide users with a more comfortable, safe, and intelligent audio experience, while promoting scientific and technological progress and sustainable development.

[0012] In the prior art CN117278905A, a sound system and a control method thereof are disclosed, including a main control system and a joint control system, wherein the main control system includes a software module and a hardware module, wherein the hardware module includes a Bluetooth BLE module, a human-computer interaction display module and a remote control box, wherein the Bluetooth BLE module is arranged in the sound, the human-computer interaction display module is arranged on the sound for displaying sound parameters, and the remote control box is used to establish a communication control connection with the Bluetooth BLE module; the present invention sets the Bluetooth BLE module as the communication connection hardware, connects multiple groups of sound systems in series through the joint control connection module, judges the strength of the connection signal between the multiple groups of sound systems and the remote control box based on the optimal channel through the competition screening module, selects the master control sound system, the remote control box controls the master control sound system, and the master control sound system joint controls other series connected sound systems, only one remote control box is needed, the operation is convenient, the multiple groups of sound systems are controlled in series, the delay is reduced, and the synchronization is improved, but the method is only applicable to a sound system consisting of multiple groups of sound systems in series, and is not applicable to the control of a single sound system. In the prior art CN 118042355A, an automatic control system for stage intelligent voice-controlled sound is disclosed, which includes a sound recording module, which is configured in the control system, and relies on the sound recording module to recognize and record the voice of the stage controller, and distinguish the daily language and control instructions of the stage controller; a visual recognition module, which is used to record the facial information of the stage controller, and after the recording is completed, connect the cameras around the stage, demarcate a preset control area in the stage area, and perform visual detection on the control area. The parameters of the stage controller are recorded and distinguished by the sound recording module and the visual recognition module, so as to avoid the stage controller triggering the control instruction during the routine conversation or the non-authorized object, resulting in the control adjustment of the sound, improve the intelligent control effect of the sound, and reduce the probability of stage performance accidents. However, this method only avoids accidents caused by the control adjustment of the sound by the command of the non-authorized person through authority control. However, this method only determines whether the instruction is issued by the authorized person through voice and visual recognition, and does not add semantic recognition processing, and cannot avoid the accidental control of the sound by the authorized person when the control instruction is involved in the routine conversation. Summary of the invention

[0013] In order to solve the above problems, the present invention proposes an adaptive sound control method and system to solve the problems existing in the above-mentioned prior art, improve the adaptive control capability of sound, especially the sound of large-scale live broadcasts such as concerts, music festivals, concert halls, and recording studios, reduce unexpected risks, and improve the on-site user experience.

[0014] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an acoustic adaptive control method, applied to an acoustic adaptive control system, including an acoustic abnormal noise adaptive filtering method, and an adaptive control method based on visual and semantic dual recognition.

[0015] Furthermore, the described method for adaptive sound control and the described method for adaptively filtering abnormal sound noise include a method for intelligently identifying noise types and a method for adaptively filtering abnormal noise, which are mainly used to identify abnormal noise signals caused by environmental, human or mechanical failure factors in large live sound systems such as concerts, music festivals, concert halls and recording studios, and automatically analyze noise categories and detailed parameters, ultimately achieving the purpose of adaptively filtering noise signals.

[0016] Furthermore, the acoustic adaptive control method and the noise type intelligent identification method convert the time-frequency signal of the sound signal into an amplitude-time curve image, accurately identify and remove the noise signal with the help of deep learning technology, and lay the foundation for noise signal filtering. The specific steps include: Step 1: Use a microphone or other audio recording equipment to capture and intercept various typical abnormal sound noises from artificial synthesis or on-site sound collection, including digital distortion noise, white noise, feedback howling, crackling noise, audience noise and environmental noise; Step 2: Use a noise analyzer, a spectrum analyzer, a sound level meter and professional computing equipment to obtain the amplitude, phase, frequency, sound level, sound pressure level, sound intensity level, sound power level, noise mean and noise variance data of various noise signals collected in step 1; Step 3: Decompose the audio signal into different frequency components. The noise signal collected in step 1 is the time domain signal of the noise. The noise time domain signal is converted into a frequency domain signal through fast Fourier transform, thereby reflecting the intensity and distribution of the noise signal at different frequencies. Step 4: Extract the amplitude information of each frequency component from the decomposed signal; Step 5: Draw a curve graph of the change in amplitude over time, where the horizontal axis represents time and the vertical axis represents amplitude; Step 6: Mark the parameter value corresponding to the noise obtained in step 2 as prior information in the acoustic signal amplitude-time curve diagram directly or indirectly through a functional relationship with the amplitude and frequency; Step 7: Place the sound signal curve graphs processed in step 6 according to categories, and divide the training set, test set, and validation set into proportions in each category of image data; Step 8: Construct a deep neural network model for image recognition. The model contains two branches, one for identifying and outputting the noise type, and the other for outputting the noise signal parameters; Step 9: Train the deep neural network model constructed in step 8 on the noisy training data set, and observe whether the loss function and accuracy parameter changes of the model on the training set and the validation set are normal during the training process. When both parameters do not change, the model is considered to have converged and the training process is completed. After that, the performance test of the trained neural network model is completed on the test set.

[0017] Furthermore, the control method of the acoustic adaptation, the abnormal noise adaptive filtering method combines the blind source separation technology and the noise filtering technology, and completes the abnormal noise filtering on the basis of the above-mentioned noise type intelligent identification method, and the specific steps include: Step 1: Obtain a mixed sound signal including an effective sound signal and various noise source signals through a microphone or other sound signal input device; Step 2: Set is a set of received N dimensional noise-free effective source signal, is the dimensional mixing matrix, is a set of received M dimensional mixed sound signal containing effective sound signal and various noises. is noise, generally considered as additive noise, then the received mixed sound signal Depend on N Unknown independent source signal and the mixing matrix A The model of linear instantaneous mixture can be expressed as:

[0018] The first step of the abnormal noise adaptive filtering method is to use blind source separation technology to separate the effective signal from various noise source signals from the mixed sound signal, that is, to obtain the separation matrix by some method. W , so as to restore each source signal through the mixed sound signal. This process can be mathematically modeled as:

[0019] Among them, is called the separated signal, and is the estimated vector of the source signal. N = M It is called positive blind source separation when N < M It is called overdetermined blind source separation when N > M It is called underdetermined blind source separation.

[0020] Step 3: Perform mean processing, also called decentralization, on the mixed sound signal obtained in step 1 to ensure that the mixed sound signal is a vector with zero mean. The specific operation method is to use the mixed sound signal Subtract its mean vector :

[0021] in, is the mathematical expectation of the mixed sound signal to be separated. The mathematical expectation of a finite number of samples can be replaced by the average value, so the above formula can be converted to

[0022] in, T the number of samples collected for the mixed acoustic signal; Step 4: Whiten the mixed sound signal, also called sphericalization, through the whitening matrix To sphericalize the received mixed sound signal, use Decorrelation, that is, The covariance matrix of is the unit matrix, so the key to this step is to find the whitening matrix , the specific operations are: Mixed sound signal after de-meaning and normalization The covariance matrix of

[0023] The main purpose of the whitening operation is to find a linear transformation that converts the positive definite covariance matrix C Convert to diagonal matrix; whiten matrix It can be obtained by performing singular value decomposition on the covariance matrix. The specific process is: The covariance matrix C Perform singular value decomposition to obtain:

[0024] in, P is an orthogonal matrix, so ; Assume that the signal after whitening operation is , then the signal The covariance matrix can be expressed as:

[0025] The formula Substituting into

[0026] In the above formula, the matrix The elements on the main diagonal of the matrix The characteristic group of , and all are greater than zero. Therefore, the above formula can be further transformed into

[0027] Assumptions ,but , so the whitening matrix Finally, it can be expressed as

[0028] in Represents the identity matrix.

[0029] Step 5: Setting the objective function. Negative entropy and kurtosis are commonly used signal statistical independence measurement indicators in blind source separation technology. However, in practical applications, kurtosis is extremely sensitive to outliers, which easily leads to unstable judgment results. Therefore, the negative entropy of the present invention is more widely used; however, the negative entropy is not easy to be accurately calculated. Therefore, the present invention adopts an improved negative entropy substitution function, and the calculation formula is:

[0030] in, Is a separation signal y No. i component signals, k j yes No. j Assume that The probability distribution of is symmetric, then , then the negative entropy is:

[0031] in, Known as The kurtosis of the negative entropy can be analyzed. Under certain conditions, there is a mapping relationship between negative entropy and kurtosis. k The expression of 4 is

[0032] Therefore, the blind source separation problem of mixed sound signals is transformed into the problem of finding the global optimal value of the objective function. The better the optimization result of the objective function is, the higher the degree of independence between the separated source sound signals is, and the separation matrix is W The better the effect; Step 6: Adopt the blind source separation objective function optimization method based on the firefly optimization algorithm. The firefly algorithm first randomly distributes a number of firefly individuals in the problem solution space. Each firefly individual represents a solution in the objective function space and has a corresponding fluorescein value at a certain moment. The size of the fluorescein value is used to measure the quality of the individual's position, that is, the fluorescein value is related to the fitness of the solution. Each firefly individual has its own corresponding decision domain at a certain moment, and the size of the decision domain is determined by the decision radius. In the decision domain, the firefly individual looks for all individuals with a larger fluorescein value than itself to form neighbors, and then moves toward the neighbors with a larger fluorescein value than itself in the direction of high probability of movement. Finally, the decision radius is updated and the process is repeated until a certain number of iterations or accuracy requirements are reached. At this time, all firefly individuals gather around brighter fireflies, that is, multiple local extreme points or global extreme points of the objective function are searched, so as to achieve the purpose of optimizing in the solution space. The specific steps include: 1) Fireflies are generated randomly n Separation matrix as the firefly group, randomly initialize the firefly individuals in the search space i Initial position and the initial decision radius ; 2) Update the fluorescein value. The higher the fluorescein value of a firefly individual, the stronger its attraction is, and the greater the probability that other individuals will move towards it. The fluorescein value is related to the fitness function fitness, and its mathematical expression is

[0033] Where, is the fluorescein value of the individual firefly; is the decay factor of the fluorescein value; is the enhancement factor of fluorescein; For Firefly i exist M The location of the dimensional real number space; For individuals The fitness value corresponding to the position; 3) Neighbor selection, individual fireflies i Find other fireflies with higher fluorescein values ​​within its perception range to form a neighbor set (neighborhood), expressed as:

[0034] in, For firefly individuals i and j The distance between For individuals i Decision radius; 4) Movement probability calculation, based on the determined neighbor set Calculating Roulette Probabilities , compare each , select the large one as the target object:

[0035] 5) Position update, based on movement probability , firefly individual i To update the position, the calculation expression is:

[0036] in, represents the Euclidean distance, Represents the flight span of a firefly.

[0037] 6) Decision radius update, firefly individual i Based on neighbor set Update decision radius , the calculation expression is:

[0038] in, is the regulating factor, is the normal perception radius of a firefly; is the neighborhood threshold; 7) Repeat steps 2) to 6) to gradually update the position and decision radius of the fireflies at the next moment. After each update, record the fitness value of the current firefly swarm. If the current fitness value is greater than the fitness value of the previous generation, save the current firefly position and fitness value as the best position and the best fitness value. Otherwise, save the previous generation result as the best position and the best fitness value. 8) If the output reaches the required separation accuracy, or the program meets the end condition, the above steps stop executing, and the optimal position and optimal fitness are output. The optimal fitness is the optimal value of the objective function in step 5, and the optimal position is the value containing the separation matrix W The optimal values ​​of all parameters at this time; Step 7: Send each independent sound signal separated from step 2 to step 6 to the noise type intelligent identification method to accurately identify the noise type and related parameters, laying a foundation for noise signal filtering; Step 8: Based on the identified noise type and its parameter characteristics such as amplitude and frequency, a dynamically adjustable bandpass filter is designed to filter out the independent noise signal separated from the mixed sound signal and retain only the effective sound signal, thereby achieving the goal of filtering out abnormal noise and outputting a pure and effective sound signal.

[0039] Furthermore, the adaptive control method for audio, the adaptive control method based on visual semantic dual recognition mainly provides an intelligent audio control method, so that the audio control personnel can still quickly control the audio even at a certain distance from the audio, and can also facilitate an audio commissioning engineer to quickly operate multiple audios at the same time, reduce the demand for backstage personnel, and improve the efficiency of audio control, including voice control and gesture control modes, the specific steps of voice control include: Step 1: Obtaining operation permissions, using deep learning-based face and voice dual recognition technology, using facial images and voice as keys to unlock operation permissions, and the sound administrator collects the facial data and voice data of operators who meet the operation permissions, and stores the two types of data in the database after marking them for deep neural network matching; Step 2: Operation authority matching. When the operator needs to operate the audio equipment, the face and voice information of the person to be authenticated can be obtained through the face and voice authentication system, and the obtained information can be matched with the face and voice data stored in the database in advance. Only after both are matched can the audio be operated. The double verification method is used to ensure the identity and authority of the operator, and prevent unauthorized personnel from obtaining audio operation authority and causing accident risks; Step 3: The present invention provides an audio intelligent voice control method, which is designed with two parts: voice-text conversion and semantic understanding. First, a voice-text database is constructed, and commonly used voice control instructions and corresponding texts are stored in the database in the form of data pairs. To ensure the quantity scale and diversity of the database, instructions with the same meaning can be expressed in sentences with different structures, voices of different genders, different languages, and dialects. Step 4: Preprocess the data set constructed in step 3, including noise reduction, echo removal, and normalization of the sound signal, as well as confirmation and correction of the matching degree between the text content and the voice and the typos in the text; Step 5: Build a neural network model for speech-to-text conversion, deploy the neural network model on the data set processed in step 4 to complete neural model training and testing, and save the weights of the neural network model with good performance; Step 6: Take out the text data in the speech-text database constructed in step 3 and construct a text database separately; Step 7: Preprocess the text data in step 6, including word segmentation, stop word removal, stemming, and part-of-speech tagging; Step 8: Use word embedding method to convert the text data processed in step 7 into a feature representation that can be processed by a machine, that is, map the words into a low-dimensional vector space and retain the semantic relationship between the words; Step 9: Build a Transformer neural network model for semantic understanding, use the preprocessed data in step 8 to train the neural network model, and optimize the performance of the model on the semantic understanding task by adjusting the model parameters.

[0040] Step 10: Perform performance test and evaluation on the trained neural network model on the test data set to verify its performance on the test set; Step 11: Optimize the model performance according to the evaluation results, including adjusting the model structure and hyperparameters, and saving the adjusted model weight parameters; Step 12: The audio operator's voice information is respectively passed through a neural network model for realizing speech-to-text translation and a neural network model for realizing semantic understanding, and the control module is called according to the understood instruction semantics to realize intelligent voice control of the audio.

[0041] The types and contents of command controls that can be realized by the voice control mode are relatively rich, but the language expression takes a long time. This mode is suitable for scenes where the timeliness of audio operation is not high. In order to meet the requirements of audio intelligent control in scenes with high timeliness of audio operation and frequent audio control operations, the present invention also provides a gesture control mode, and the specific steps include: Step 1: Build a gesture recognition database, store various gesture images and their corresponding command meanings into data pairs in the database, where the gesture image is the image to be recognized and the command meaning is the corresponding label. To improve the generalization ability of the neural network model, the database needs to contain gesture images of different angles, lighting conditions, background complexity, and human hand skin color; Step 2: Preprocess the image data in the database constructed in step 1, including cropping, scaling, and normalization operations to eliminate redundant information and noise in the image; Step 3: Use skin color detection and edge detection technology to extract the hand area and reduce background interference; Step 4: Build a deep neural network model, and complete the training and testing of the deep neural network model on the labeled gesture data processed in steps 2 to 3. Adjust the model parameters through the back propagation algorithm to minimize the difference between the predicted label and the actual label. Step 6: Use the trained neural network model to complete the recognition of the gesture command in step 3, and implement adaptive control of the audio according to the command corresponding to the gesture.

[0042] Furthermore, in order to prevent the operator's language and gestures from causing misoperation of the audio system, the present invention designs an optional control mechanism, including a voice wake-up and occlusion delay mechanism. In the voice wake-up mode, the operator wakes up the audio operating system through a pre-set wake-up voice. The operating system will require the operator to confirm the operation twice. Only after the confirmation is passed will the subsequent voice control instructions be executed; the delayed occlusion mechanism sets a 3-second occlusion delay time. When the camera detects that there is an obstruction within 50 cm in front for more than 3 seconds, the system considers that the operator is applying to start the gesture recognition control mode, and automatically turns on the camera to capture gesture data and sends it to the neural network model for gesture recognition. If no valid gesture signal is detected, it is considered that a false touch has occurred, and the control system automatically enters a sleep state.

[0043] On the other hand, a sound adaptive control system is provided for use in any one of the sound adaptive control methods, the sound adaptive control system comprising: Mixed sound signal acquisition subsystem: It is composed of a single microphone, an array microphone and a matching professional sound signal acquisition and receiving device. It is used to collect effective sound signals and environmental noise data generated by the external environment, including background noise, human voice noise, etc. It is an important data source support subsystem of the present invention; Audio status detection and operation terminal: The audio status detection and operation terminal is located on a computer or server device, and includes a host computer software display interface and a hardware support module. The host computer software display interface is located on the computer device monitor, and can display the current internal mixed sound signal of the audio, the waveform and related parameters of each independent sound signal after blind source separation in real time, and provide the option of manual operation to control the audio; the hardware support module includes a camera module, an audio signal acquisition module, a central processing unit and a GPU computing power module. The camera module and the audio signal acquisition module are used to realize user identity authentication, voice control and gesture control. The central processing unit overall processes all data in the subsystem, and the GPU computing power module provides sufficient computing power support for neural network model training and deployment. This part is the core of the audio adaptive control system; Noise adaptive filtering subsystem: This subsystem has a built-in adaptive filtering method for abnormal acoustic noise, including two modules: noise type recognition and noise adaptive filtering. The noise type recognition module has a built-in data preprocessing method and a neural network model, which can realize the data conversion from acoustic signals to acoustic signal waveform images. The neural network model can complete the identification of noise signal types and key parameters with the support of GPU computing power; the noise adaptive filtering module relies on adaptive variable bandpass filters and blind source separation technology to achieve noise separation and filtering; Adaptive control subsystem with visual and semantic dual recognition: This subsystem has a built-in adaptive control method based on visual and semantic dual recognition, and supports dual control methods of voice and gesture through two neural network models for performing semantic understanding and gesture recognition respectively.

[0044] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention proposes a method for adaptively filtering abnormal sound noise. By combining deep learning, blind source separation, bandpass filter and multiple data preprocessing methods, it can realize adaptive identification, separation and filtering of multiple mixed noises that may be contained in the sound. It can realize adaptive noise filtering that is mistakenly considered to be involved in a noisy noise environment, and effectively improve the quality of the audio signal output by the sound.

[0045] 2. The present invention proposes an adaptive control method based on visual and semantic dual recognition, which ensures the absolute operating authority of the sound operator through face and voice dual authentication, preventing unauthorized personnel from operating the sound without authorization and causing stage accidents or risks; designs two control modes, voice and gesture, to improve the control efficiency of the sound, reduce the demand for backstage personnel, and reduce labor costs.

[0046] 3. The present invention provides an adaptive sound control system, which provides effective hardware support for the above two methods, transforms theoretical derivation and software simulation into actual hardware products, and improves the feasibility of the theory. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of the sound adaptation control method of the present invention; Figure 2 This is a structural diagram of the sound adaptive control system of the present invention; Figure 3 This is a flow chart of the firefly optimization algorithm of the present invention; DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0049] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0050] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0051] In an embodiment of the present invention, a method for controlling sound adaptation is provided. Figure 1 , applied to acoustic adaptive control systems, including an adaptive filtering method for acoustic abnormal noise and an adaptive control method based on visual and semantic dual recognition.

[0052] The method for adaptively filtering abnormal sound noise includes an intelligent noise type recognition method and an abnormal noise adaptive filtering method. It is mainly used to identify abnormal noise signals caused by environmental, human or mechanical failure factors in large live sound systems such as concerts, music festivals, concert halls, and recording studios, and automatically analyze noise categories and detailed parameters to ultimately achieve the purpose of adaptively filtering noise signals.

[0053] The adaptive control method based on visual and semantic dual recognition takes into account both the security mechanism and efficiency of audio operation, and provides a face and voice dual recognition authentication mechanism to ensure that the operator who can control the audio is a legally authorized operator. At the same time, it provides a voice and gesture dual recognition audio control solution, which can support intelligent voice control under normal conditions and quick gesture command control in special scenarios, greatly improving the efficiency of intelligent control of the audio.

[0054] Refer to the attached Figure 2 The present invention also provides an adaptive sound control system, including a mixed sound signal acquisition subsystem, an audio state detection and operation terminal, a noise adaptive filtering subsystem and a visual and semantic dual recognition adaptive control subsystem.

[0055] The hybrid sound signal acquisition subsystem is composed of a single microphone, an array microphone and a matching professional sound signal acquisition and receiving device. It is used to collect effective sound signals and environmental noise data generated by the external environment, including background noise, human voice noise, etc. It is an important data source support subsystem of the present invention; The audio status detection and operation terminal is located on a computer or server device, and includes a host computer software display interface and a hardware support module. The host computer software display interface is located on the computer device display, and can display the current internal mixed sound signal of the audio, the waveform and related parameters of each independent sound signal after blind source separation in real time, and provide the option of manual operation to control the audio; the hardware support module includes a camera module, an audio signal acquisition module, a central processing unit and a GPU computing power module. The camera module and the audio signal acquisition module are used to realize user identity authentication, voice control and gesture control. The central processing unit overall processes all data in the subsystem, and the GPU computing power module provides sufficient computing power support for neural network model training and deployment. This part is the core of the audio adaptive control system; The noise adaptive filtering subsystem has a built-in abnormal acoustic noise adaptive filtering method, including two modules: noise type recognition and noise adaptive filtering. The noise type recognition module has a built-in data preprocessing method and a neural network model, which can realize the data conversion from acoustic signal to acoustic signal waveform image. The neural network model can complete the recognition of noise signal types and key parameters with the support of GPU computing power; the noise adaptive filtering module relies on adaptive variable bandpass filter and blind source separation technology to realize noise separation and filtering; The adaptive control subsystem of visual and semantic dual recognition has a built-in adaptive control method based on visual and semantic dual recognition, and supports dual control modes of voice and gesture through two neural network models for performing semantic understanding and gesture recognition respectively.

[0056] In a specific embodiment, the sound engineers of large live broadcasts such as concerts, music festivals, concert halls, and recording studios collect live sound signals through a mixed sound signal acquisition subsystem. The sound signals include effective human voices, background music sound signals, and on-site environmental noise. The collected mixed sound signals are then mixed with the mechanical noise and current noise inside the sound system to form the final mixed sound signal. After obtaining the mixed sound signal, the sound engineer sends the mixed sound signal to the noise adaptive filtering subsystem, and uses the built-in noise adaptive filtering method to demix the mixed sound signal into independent effective sound signals and noise signals from different sources. After that, the noise type identification module accurately identifies the noise type and related parameters. Finally, a bandpass filter is designed based on the identified noise type and parameters to filter out the noise signal and output a pure and effective sound signal. During the use of the audio system, the audio debugger monitors the audio status through the audio status detection and operation terminal, including the output signal waveform and audio parameter values, and realizes the control of the audio system. The specific process includes first turning on the camera module of the audio status detection and operation terminal to complete the facial information verification, and then completing the sound signal verification through the audio signal module. After two verifications, the operation authority is obtained, and then the adaptive control method based on visual semantic dual recognition built into the adaptive control subsystem of visual semantic dual recognition realizes convenient voice or gesture control of the audio system. In order to prevent accidental touches, the debugger can also turn on the anti-mistouch mode and perform anti-mistouch operations through voice wake-up mechanism or delayed occlusion mechanism.

[0057] Example 1

[0058] In one embodiment, the sound engineer performs noise reduction processing on the mixed sound signal in the sound equipment through the sound abnormal noise adaptive filtering method to ensure the sound quality of the sound output. The sound abnormal noise adaptive filtering method includes a noise type intelligent identification method and an abnormal noise adaptive filtering method, which is mainly used to identify abnormal noise signals caused by environmental, human or mechanical failure factors in large live sound such as concerts, music festivals, concert halls, and recording halls, and automatically analyze the noise category and detailed parameters, and finally achieve the purpose of adaptively filtering the noise signal. Among them, the noise type intelligent identification method converts the time-frequency signal of the sound signal into an amplitude-time curve image, and accurately identifies and removes the noise signal with the help of deep learning technology, laying a foundation for noise signal filtering. The specific steps include: Step 1: Use a microphone or other audio recording equipment to capture and intercept various typical abnormal sound noises from artificial synthesis or on-site sound collection, including digital distortion noise, white noise, feedback howling, crackling noise, audience noise and environmental noise; Step 2: Use a noise analyzer, a spectrum analyzer, a sound level meter and professional computing equipment to obtain the amplitude, phase, frequency, sound level, sound pressure level, sound intensity level, sound power level, noise mean and noise variance data of various noise signals collected in step 1; Step 3: Decompose the audio signal into different frequency components. The noise signal collected in step 1 is the time domain signal of the noise. The noise time domain signal is converted into a frequency domain signal through fast Fourier transform, thereby reflecting the intensity and distribution of the noise signal at different frequencies. Step 4: Extract the amplitude information of each frequency component from the decomposed signal; Step 5: Draw a curve graph of the change in amplitude over time, where the horizontal axis represents time and the vertical axis represents amplitude; Step 6: Mark the parameter value corresponding to the noise obtained in step 2 as prior information in the acoustic signal amplitude-time curve diagram directly or indirectly through a functional relationship with the amplitude and frequency; Step 7: Place the sound signal curve graphs processed in step 6 according to categories, and divide the training set, test set, and validation set into proportions in each category of image data; Step 8: Construct a deep neural network model for image recognition. The model contains two branches, one for identifying and outputting the noise type, and the other for outputting the noise signal parameters; Step 9: Train the deep neural network model constructed in step 8 on the noisy training data set, and observe whether the loss function and accuracy parameter changes of the model on the training set and the validation set are normal during the training process. When both parameters do not change, the model is considered to have converged and the training process is completed. After that, the performance test of the trained neural network model is completed on the test set.

[0059] Optionally, the noise performance parameters that need to be measured and calculated in the above step 2 can be selectively measured and calculated according to the actual hardware equipment conditions on site or other requirements. Generally speaking, within a certain range, the more noise performance parameters are measured and calculated, the more prior knowledge the neural network acquires, the more comprehensive the understanding of noise performance, and the better the performance of the neural network model.

[0060] Optionally, the curve graph drawn in the above step 5 may also be a frequency-amplitude curve graph with frequency as the horizontal axis and amplitude as the vertical axis.

[0061] In the above step eight, the two output branches of the neural network use independent optimization objective functions and measurement indicators respectively, so that the two branches can be trained separately without affecting each other. The training is considered successful only when the test results of the measurement indicators of the two branches meet the standards.

[0062] The abnormal noise adaptive filtering method combines the blind source separation technology and the noise filtering technology to complete the abnormal noise filtering on the basis of the above-mentioned noise type intelligent identification method. The specific steps include: Step 1: Obtain a mixed sound signal including an effective sound signal and various noise source signals through a microphone or other sound signal input device; Step 2: Set is a set of received N dimensional noise-free effective source signal, is the dimensional mixing matrix, is a set of received M dimensional mixed sound signal containing effective sound signal and various noises. is noise, generally considered as additive noise, then the received mixed sound signal Depend on N Unknown independent source signal and the mixing matrix A The model of linear instantaneous mixture can be expressed as:

[0063] The first step of the abnormal noise adaptive filtering method is to use blind source separation technology to separate the effective signal from various noise source signals from the mixed sound signal, that is, to obtain the separation matrix by some method. W , so as to restore each source signal through the mixed sound signal. This process can be mathematically modeled as:

[0064] Among them, is called the separated signal, and is the estimated vector of the source signal. N = M When is called positive blind source separation; when N < M It is called overdetermined blind source separation when N > M It is called underdetermined blind source separation.

[0065] Step 3: Perform mean processing, also called decentralization, on the mixed sound signal obtained in step 1 to ensure that the mixed sound signal is a vector with zero mean. The specific operation method is to use the mixed sound signal Subtract its mean vector :

[0066] in, is the mathematical expectation of the mixed sound signal to be separated. The mathematical expectation of a finite number of samples can be replaced by the average value, so the above formula can be converted to

[0067] in, T the number of samples collected for the mixed acoustic signal; Step 4: Whiten the mixed sound signal, also called sphericalization, through the whitening matrix To spherize the received mixed sound signal, use Decorrelation, that is, The covariance matrix of is the unit matrix, so the key to this step is to find the whitening matrix , the specific operations are: Mixed sound signal after de-meaning and normalization The covariance matrix of

[0068] The main purpose of the whitening operation is to find a linear transformation that converts the positive definite covariance matrix C Convert to diagonal matrix; whiten matrix It can be obtained by performing singular value decomposition on the covariance matrix. The specific process is: The covariance matrix C Perform singular value decomposition to obtain:

[0069] in, P is an orthogonal matrix, so ; Assume that the signal after whitening operation is , then the signal The covariance matrix can be expressed as:

[0070] The formula Substituting into

[0071] In the above formula, the matrix The elements on the main diagonal of the matrix The characteristic group of , and all are greater than zero. Therefore, the above formula can be further transformed into

[0072] Assumptions ,but , so the whitening matrix Finally, it can be expressed as

[0073] in Represents the identity matrix.

[0074] Step 5: Setting the objective function. Negative entropy and kurtosis are commonly used signal statistical independence measurement indicators in blind source separation technology. However, in practical applications, kurtosis is extremely sensitive to outliers, which easily leads to unstable judgment results. Therefore, the negative entropy of the present invention is more widely used; however, the negative entropy is not easy to be accurately calculated. Therefore, the present invention adopts an improved negative entropy substitution function, and the calculation formula is:

[0075] in, Is the separation signal y No. i component signals, k j yes No. j Assume that The probability distribution of is symmetric, then , then the negative entropy is:

[0076] in, Known as The kurtosis of the negative entropy can be analyzed. Under certain conditions, there is a mapping relationship between negative entropy and kurtosis. k The expression of 4 is

[0077] Therefore, the blind source separation problem of mixed sound signals is transformed into the problem of finding the global optimal value of the objective function. The better the optimization result of the objective function is, the higher the degree of independence between the separated source sound signals is, and the separation matrix is W The better the effect; Step 6: Adopt the blind source separation objective function optimization method based on the firefly optimization algorithm. The firefly algorithm first randomly distributes a number of firefly individuals in the problem solution space. Each firefly individual represents a solution in the objective function space and has a corresponding fluorescein value at a certain moment. The size of the fluorescein value is used to measure the quality of the individual's position, that is, the fluorescein value is related to the fitness of the solution. Each firefly individual has its own corresponding decision domain at a certain moment, and the size of the decision domain is determined by the decision radius. In the decision domain, the firefly individual looks for all individuals with a larger fluorescein value than itself to form neighbors, and then moves toward the neighbors with a larger fluorescein value than itself in the direction of high probability of movement. Finally, the decision radius is updated and the process is repeated until a certain number of iterations or accuracy requirements are reached. At this time, all firefly individuals gather around brighter fireflies, that is, multiple local extreme points or global extreme points of the objective function are searched, so as to achieve the purpose of optimizing in the solution space. The specific steps include: 1) Fireflies are generated randomly nSeparation matrix as the firefly group, randomly initialize the firefly individuals in the search space i Initial position and the initial decision radius ; 2) Update the fluorescein value. The higher the fluorescein value of a firefly individual, the stronger its attraction is, and the greater the probability that other individuals will move towards it. The fluorescein value is related to the fitness function fitness, and its mathematical expression is

[0078] Where, is the fluorescein value of the individual firefly; is the decay factor of the fluorescein value; is the enhancement factor of fluorescein; For Firefly i exist M The location of the dimensional real number space; For individuals The fitness value corresponding to the position; 3) Neighbor selection, individual fireflies i Find other fireflies with higher fluorescein values ​​within its perception range to form a neighbor set (neighborhood), expressed as:

[0079] in, For firefly individuals i and j The distance between For individuals i Decision radius; 4) Movement probability calculation, based on the determined neighbor set Calculating Roulette Probabilities , compare each , select the large one as the target object:

[0080] 5) Position update, based on movement probability , firefly individual i To update the position, the calculation expression is:

[0081] in, represents the Euclidean distance, Represents the flight span of a firefly.

[0082] 6) Decision radius update, firefly individual i Based on neighbor set Update decision radius , the calculation expression is:

[0083] in, is the regulating factor, is the normal perception radius of a firefly; is the neighborhood threshold; 7) Repeat steps 2) to 6) to gradually update the position and decision radius of the fireflies at the next moment. After each update, record the fitness value of the current firefly swarm. If the current fitness value is greater than the fitness value of the previous generation, save the current firefly position and fitness value as the best position and the best fitness value. Otherwise, save the previous generation result as the best position and the best fitness value. 8) If the output reaches the required separation accuracy, or the program meets the end condition, the above steps stop executing, and the optimal position and optimal fitness are output. The optimal fitness is the optimal value of the objective function in step 5, and the optimal position is the value containing the separation matrix W The optimal values ​​of all parameters at this time; Step 7: Send each independent sound signal separated from step 2 to step 6 to the noise type intelligent identification method to accurately identify the noise type and related parameters, laying a foundation for noise signal filtering; Step 8: Based on the identified noise type and its parameter characteristics such as amplitude and frequency, a dynamically adjustable bandpass filter is designed to filter out the independent noise signal separated from the mixed sound signal and retain only the effective sound signal, thereby achieving the goal of filtering out abnormal noise and outputting a pure and effective sound signal.

[0084] The above firefly optimization algorithm is a global optimization algorithm. The specific algorithm flow can be found in the attached Figure 3 .

[0085] Example 2

[0086] In one embodiment, the sound engineer implements intelligent adaptive control of the sound system through an adaptive control method based on visual semantic dual recognition, so that the sound system can be quickly controlled at a certain distance from the sound system. It can also facilitate a sound engineer to quickly operate multiple sound systems at the same time, reduce the demand for backstage personnel, and improve the efficiency of sound control. The two modes are voice control and gesture control. The specific steps of voice control include: Step 1: Obtaining operation permissions, using deep learning-based face and voice dual recognition technology, using facial images and voice as keys to unlock operation permissions, and the sound administrator collects the facial data and voice data of operators who meet the operation permissions, and stores the two types of data in the database after marking them for deep neural network matching; Step 2: Operation authority matching. When the operator needs to operate the audio equipment, the face and voice information of the person to be authenticated can be obtained through the face and voice authentication system, and the obtained information can be matched with the face and voice data stored in the database in advance. Only after both are matched can the audio be operated. The double verification method is used to ensure the identity and authority of the operator, and prevent unauthorized personnel from obtaining audio operation authority and causing accident risks; Step 3: The present invention provides an audio intelligent voice control method, which is designed with two parts: voice-text conversion and semantic understanding. First, a voice-text database is constructed, and commonly used voice control instructions and corresponding texts are stored in the database in the form of data pairs. To ensure the quantity scale and diversity of the database, instructions with the same meaning can be expressed in sentences with different structures, voices of different genders, different languages, and dialects. Step 4: Preprocess the data set constructed in step 3, including noise reduction, echo removal, and normalization of the sound signal, as well as confirmation and correction of the matching degree between the text content and the voice and the typos in the text; Step 5: Build a neural network model for speech-to-text conversion, deploy the neural network model on the data set processed in step 4 to complete neural model training and testing, and save the weights of the neural network model with good performance; Step 6: Take out the text data in the speech-text database constructed in step 3 and construct a text database separately; Step 7: Preprocess the text data in step 6, including word segmentation, stop word removal, stemming, and part-of-speech tagging; Step 8: Use word embedding method to convert the text data processed in step 7 into a feature representation that can be processed by a machine, that is, map the words into a low-dimensional vector space and retain the semantic relationship between the words; Step 9: Build a Transformer neural network model for semantic understanding, use the preprocessed data in step 8 to train the neural network model, and optimize the performance of the model on the semantic understanding task by adjusting the model parameters.

[0087] Step 10: Perform performance test and evaluation on the trained neural network model on the test data set to verify its performance on the test set; Step 11: Optimize the model performance according to the evaluation results, including adjusting the model structure and hyperparameters, and saving the adjusted model weight parameters; Step 12: The audio operator's voice information is respectively passed through a neural network model for realizing speech-to-text translation and a neural network model for realizing semantic understanding, and the control module is called according to the understood instruction semantics to realize intelligent voice control of the audio.

[0088] The types and contents of command controls that can be realized by the voice control mode are relatively rich, but the language expression takes a long time. This mode is suitable for scenes where the timeliness of audio operation is not high. In order to meet the requirements of audio intelligent control in scenes with high timeliness of audio operation and frequent audio control operations, the present invention also provides a gesture control mode, and the specific steps include: Step 1: Build a gesture recognition database, store various gesture images and their corresponding command meanings into data pairs in the database, where the gesture image is the image to be recognized and the command meaning is the corresponding label. To improve the generalization ability of the neural network model, the database needs to contain gesture images of different angles, lighting conditions, background complexity, and human hand skin color; Step 2: Preprocess the image data in the database constructed in step 1, including cropping, scaling, and normalization operations to eliminate redundant information and noise in the image; Step 3: Use skin color detection and edge detection technology to extract the hand area and reduce background interference; Step 4: Build a deep neural network model, and complete the training and testing of the deep neural network model on the labeled gesture data processed in steps 2 to 3. Adjust the model parameters through the back propagation algorithm to minimize the difference between the predicted label and the actual label. Step 6: Use the trained neural network model to complete the recognition of the gesture command in step 3, and implement adaptive control of the audio according to the command corresponding to the gesture.

[0089] Optionally, to prevent the operator's language and gestures from causing misoperation of the audio system, the debugger may selectively use a safety control mechanism, including a voice wake-up and occlusion delay mechanism. In the voice wake-up mode, the operator wakes up the audio operating system through a pre-set wake-up voice. The operating system will require the operator to confirm the operation twice, and only after the confirmation is passed will the subsequent voice control instructions be executed; the delayed occlusion mechanism sets a 3-second occlusion delay time. When the camera detects an obstruction within 50 cm in front for more than 3 seconds, the system considers that the operator is applying to start the gesture recognition control mode, and automatically turns on the camera to capture gesture data and sends it to the neural network model for gesture recognition. If no valid gesture signal is detected, it is considered that a false touch has occurred, and the control system automatically enters sleep mode.

[0090] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0091] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An adaptive sound control method, including an adaptive sound abnormal noise filtering method, an adaptive control method based on visual and semantic dual recognition, characterized in that: The adaptive filtering method for abnormal sound noise identifies abnormal noise signals in the sound caused by environmental, human or mechanical failure factors, and automatically analyzes the noise category and detailed parameters, ultimately achieving the purpose of adaptively filtering noise signals; The adaptive control method based on visual and semantic dual recognition takes into account both the security mechanism and efficiency of audio operation, and provides a face and voice dual recognition authentication mechanism to ensure that the operator who can control the audio is a legally authorized operator. At the same time, it provides a voice and gesture dual recognition audio control solution, which can support intelligent voice control under normal conditions and quick gesture command control in special scenarios, greatly improving the efficiency of intelligent control of the audio.

2. The method for controlling acoustic adaptation according to claim 1, characterized in that: The method for adaptively filtering abnormal acoustic noise includes a method for intelligently identifying noise types and a method for adaptively filtering abnormal noise.

3. The method for controlling acoustic adaptation according to claim 2, characterized in that: The noise type intelligent identification method converts the time-frequency signal of the sound signal into an amplitude-time curve image, accurately identifies and removes the noise signal with the help of deep learning technology, and lays the foundation for noise signal filtering. The specific steps include: Step 1: Use a microphone or other audio recording equipment to capture and intercept various typical abnormal sound noises from artificial synthesis or on-site sound collection, including digital distortion noise, white noise, feedback howling, crackling noise, audience noise and environmental noise; Step 2: Use a noise analyzer, a spectrum analyzer, a sound level meter and professional computing equipment to obtain the amplitude, phase, frequency, sound level, sound pressure level, sound intensity level, sound power level, noise mean and noise variance data of various noise signals collected in step 1; Step 3: Decompose the audio signal into different frequency components. The noise signal collected in step 1 is the time domain signal of the noise. The noise time domain signal is converted into a frequency domain signal through fast Fourier transform, thereby reflecting the intensity and distribution of the noise signal at different frequencies. Step 4: Extract the amplitude information of each frequency component from the decomposed signal; Step 5: Draw a curve graph of the change in amplitude over time, where the horizontal axis represents time and the vertical axis represents amplitude; Step 6: Mark the parameter value corresponding to the noise obtained in step 2 as prior information in the acoustic signal amplitude-time curve diagram directly or indirectly through a functional relationship with the amplitude and frequency; Step 7: Place the sound signal curve graphs processed in step 6 according to categories, and divide the training set, test set, and validation set into proportions in each category of image data; Step 8: Construct a deep neural network model for image recognition. The model contains two branches, one for identifying and outputting the noise type, and the other for outputting the noise signal parameters; Step 9: Train the deep neural network model constructed in step 8 on the noisy training data set, and observe whether the loss function and accuracy parameter changes of the model on the training set and the validation set are normal during the training process. When both parameters do not change, the model is considered to have converged and the training process is completed. After that, the performance test of the trained neural network model is completed on the test set.

4. The method for controlling acoustic adaptation according to claim 2, characterized in that: The abnormal noise adaptive filtering method combines the blind source separation technology and the noise filtering technology to complete the abnormal noise filtering on the basis of the above-mentioned noise type intelligent identification method. The specific steps include: Step 1: Obtain a mixed sound signal including an effective sound signal and various noise source signals through a microphone or other sound signal input device; Step 2: Set is a set of received N dimensional noise-free effective source signal, is the dimensional mixing matrix, is a set of received M dimensional mixed sound signal containing effective sound signal and various noises. is noise, generally considered as additive noise, then the received mixed sound signal Depend on N Unknown independent source signal and the mixing matrix A The model of linear instantaneous mixture can be expressed as: The first step of the abnormal noise adaptive filtering method is to use blind source separation technology to separate the effective signal from various noise source signals from the mixed sound signal, that is, to obtain the separation matrix by some method. W , so as to restore each source signal through the mixed sound signal. This process can be mathematically modeled as: ; where is called the separated signal and is the estimated vector of the source signal. N = M It is called positive blind source separation when N < M It is called overdetermined blind source separation when N > M It is called underdetermined blind source separation; Step 3: Perform mean processing, also called decentralization, on the mixed sound signal obtained in step 1 to ensure that the mixed sound signal is a vector with zero mean. The specific operation method is to use the mixed sound signal Subtract its mean vector : ;in, is the mathematical expectation of the mixed sound signal to be separated. The mathematical expectation of a finite number of samples can be replaced by the average value, so the above formula can be converted to ;in, T the number of samples collected for the mixed acoustic signal; Step 4: Whiten the mixed sound signal, also called sphericalization, through the whitening matrix To spherize the received mixed sound signal, use Decorrelation, that is, The covariance matrix of is the unit matrix, so the key to this step is to find the whitening matrix , the specific operations are: Mixed sound signal after de-meaning and normalization The covariance matrix of ; The main purpose of the whitening operation is to find a linear transformation to convert the positive definite covariance matrix C Convert to diagonal matrix; whiten matrix It can be obtained by performing singular value decomposition on the covariance matrix. The specific process is: The covariance matrix C Perform singular value decomposition to obtain: ;in, P is an orthogonal matrix, so ; Assume that the signal after whitening operation is , then the signal The covariance matrix can be expressed as: ; The formula Substituting into ; In the above formula, the matrix The elements on the main diagonal of the matrix The characteristic group of , and all are greater than zero. Therefore, the above formula can be further transformed into Assumptions ,but , so the whitening matrix Finally, it can be expressed as ;in represents the identity matrix; Step 5: Setting the objective function. Negative entropy and kurtosis are commonly used signal statistical independence measurement indicators in blind source separation technology. However, in practical applications, kurtosis is extremely sensitive to outliers, which easily leads to unstable judgment results. Therefore, the negative entropy of the present invention is more widely used; however, the negative entropy is not easy to be accurately calculated. Therefore, the present invention adopts an improved negative entropy substitution function, and the calculation formula is: ;in, Is the separation signal y No. i component signals, k j yes No. j Assume that The probability distribution of is symmetric, then , then the negative entropy is: ;in, Known as The analysis shows that under certain conditions, there is a mapping relationship between negative entropy and kurtosis. k The expression of 4 is Therefore, the blind source separation problem of mixed sound signals is transformed into the problem of finding the global optimal value of the objective function. The better the optimization result of the objective function is, the higher the degree of independence between the separated source sound signals is, and the separation matrix W The better the effect; Step 6: Adopt the blind source separation objective function optimization method based on the firefly optimization algorithm. The firefly algorithm first randomly distributes a number of firefly individuals in the problem solution space. Each firefly individual represents a solution in the objective function space and has a corresponding fluorescein value at a certain moment. The size of the fluorescein value is used to measure the quality of the individual's position, that is, the fluorescein value is related to the fitness of the solution. Each firefly individual has its own corresponding decision domain at a certain moment, and the size of the decision domain is determined by the decision radius. In the decision domain, the firefly individual looks for all individuals with a larger fluorescein value than itself to form neighbors, and then moves toward the neighbors with a larger fluorescein value than itself in the direction of high probability of movement. Finally, the decision radius is updated and the process is repeated until a certain number of iterations or accuracy requirements are reached. At this time, all firefly individuals gather around brighter fireflies, that is, multiple local extreme points or global extreme points of the objective function are searched, so as to achieve the purpose of optimizing in the solution space. The specific steps include: 1) Fireflies are generated randomly n Separation matrix as the firefly group, randomly initialize the firefly individuals in the search space i Initial position and the initial decision radius ; 2) Update the fluorescein value. The higher the fluorescein value of a firefly individual, the stronger its attraction is, and the greater the probability that other individuals will move towards it. The fluorescein value is related to the fitness function fitness, and its mathematical expression is ; Wherein, is the fluorescein value of the individual firefly; is the decay factor of the fluorescein value; is the enhancement factor of fluorescein; For Firefly i exist M The location of the dimensional real number space; For individuals The fitness value corresponding to the position; 3) Neighbor selection, individual fireflies i Find other fireflies with higher fluorescein values ​​within its perception range to form a neighbor set (neighborhood), expressed as: ;in, For firefly individuals i and j The distance between For individuals i Decision radius; 4) Movement probability calculation, based on the determined neighbor set Calculating Roulette Probabilities , compare each , select the large one as the target object: ； 5) Position update, based on movement probability , firefly individual i To update the position, the calculation expression is: ;in, represents the Euclidean distance, represents the flight span of a firefly; 6) Decision radius update, firefly individual i Based on neighbor set Update decision radius , the calculation expression is: ;in, is the regulating factor, is the normal perception radius of a firefly; is the neighborhood threshold; 7) Repeat steps 2) to 6) to gradually update the position and decision radius of the fireflies at the next moment. After each update, record the fitness value of the current firefly swarm. If the current fitness value is greater than the fitness value of the previous generation, save the current firefly position and fitness value as the best position and the best fitness value. Otherwise, save the previous generation result as the best position and the best fitness value. 8) If the output reaches the required separation accuracy, or the program meets the end condition, the above steps stop executing, and the optimal position and optimal fitness are output. The optimal fitness is the optimal value of the objective function in step 5, and the optimal position is the value containing the separation matrix W The optimal values ​​of all parameters at this time; Step 7: Send each independent sound signal separated from step 2 to step 6 to the noise type intelligent identification method to accurately identify the noise type and related parameters, laying a foundation for noise signal filtering; Step 8: Based on the identified noise type and its parameter characteristics such as amplitude and frequency, a dynamically adjustable bandpass filter is designed to filter out the independent noise signal separated from the mixed sound signal and retain only the effective sound signal, thereby achieving the goal of filtering out abnormal noise and outputting a pure and effective sound signal.

5. The method for controlling acoustic adaptation according to claim 1, characterized in that: The adaptive control method based on visual and semantic dual recognition provides an intelligent audio control method, including two modes: voice control and gesture control.

6. The method for controlling acoustic adaptation according to claim 5, characterized in that: The specific steps of the voice control mode include: Step 1: Obtaining operation permissions, using deep learning-based face and voice dual recognition technology, using facial images and voice as keys to unlock operation permissions, and the sound administrator collects the facial data and voice data of operators who meet the operation permissions, and stores the two types of data in the database after marking them for deep neural network matching; Step 2: Operation authority matching. When the operator needs to operate the audio equipment, the face and voice information of the person to be authenticated can be obtained through the face and voice authentication system, and the obtained information can be matched with the face and voice data stored in the database in advance. Only after both are matched can the audio be operated. The double verification method is used to ensure the identity and authority of the operator, and prevent unauthorized personnel from obtaining audio operation authority and causing accident risks; Step 3: The present invention provides an audio intelligent voice control method, which is designed with two parts: voice-text conversion and semantic understanding. First, a voice-text database is constructed, and commonly used voice control instructions and corresponding texts are stored in the database in the form of data pairs. To ensure the quantity scale and diversity of the database, instructions with the same meaning can be expressed in sentences with different structures, voices of different genders, different languages, and dialects. Step 4: Preprocess the data set constructed in step 3, including noise reduction, echo removal, and normalization of the sound signal, as well as confirmation and correction of the matching degree between the text content and the voice and the typos in the text; Step 5: Build a neural network model for speech-to-text conversion, deploy the neural network model on the data set processed in step 4 to complete neural model training and testing, and save the weights of the neural network model with good performance; Step 6: Take out the text data in the speech-text database constructed in step 3 and construct a text database separately; Step 7: Preprocess the text data in step 6, including word segmentation, stop word removal, stemming, and part-of-speech tagging; Step 8: Use word embedding method to convert the text data processed in step 7 into a feature representation that can be processed by a machine, that is, map the words into a low-dimensional vector space and retain the semantic relationship between the words; Step 9: Build a Transformer neural network model for semantic understanding, use the data preprocessed in step 8 to train the neural network model, and optimize the performance of the model on the semantic understanding task by adjusting the model parameters; Step 10: Perform performance test and evaluation on the trained neural network model on the test data set to verify its performance on the test set; Step 11: Optimize the model performance according to the evaluation results, including adjusting the model structure and hyperparameters, and saving the adjusted model weight parameters; Step 12: The audio operator's voice information is respectively passed through a neural network model for realizing speech-to-text translation and a neural network model for realizing semantic understanding, and the control module is called according to the understood instruction semantics to realize intelligent voice control of the audio.

7. The method for controlling acoustic adaptation according to claim 5, characterized in that: The gesture control mode is suitable for intelligent audio control in scenarios where the audio operation timeliness is high and the audio control operation is frequent. The specific steps include: Step 1: Build a gesture recognition database, store various gesture images and their corresponding command meanings into data pairs in the database, where the gesture image is the image to be recognized and the command meaning is the corresponding label. To improve the generalization ability of the neural network model, the database needs to contain gesture images of different angles, lighting conditions, background complexity, and human hand skin color; Step 2: Preprocess the image data in the database constructed in step 1, including cropping, scaling, and normalization operations to eliminate redundant information and noise in the image; Step 3: Use skin color detection and edge detection technology to extract the hand area and reduce background interference; Step 4: Build a deep neural network model, and complete the training and testing of the deep neural network model on the labeled gesture data processed in steps 2 to 3. Adjust the model parameters through the back propagation algorithm to minimize the difference between the predicted label and the actual label. Step 6: Use the trained neural network model to complete the recognition of the gesture command in step 3, and implement adaptive control of the audio according to the command corresponding to the gesture.

8. The method for controlling sound adaptation according to claim 1, characterized in that: The adaptive control method based on visual and semantic dual recognition designs an optional control mechanism, including voice wake-up and occlusion delay mechanisms. In the voice wake-up mode, the operator wakes up the audio operating system through a pre-set wake-up voice. The operating system will require the operator to perform a second confirmation of the operation. Only after the confirmation is passed will the subsequent voice control instructions be executed; the delayed occlusion mechanism sets a 3-second occlusion delay time. When the camera detects that there is an obstruction within 50 cm in front for more than 3 seconds, the system considers that the operator is applying to start the gesture recognition control mode, and automatically turns on the camera to capture gesture data and sends it to the neural network model for gesture recognition. If no valid gesture signal is detected, it is considered that a false touch has occurred, and the control system automatically enters a sleep state.

9. An adaptive sound control system, characterized in that: The sound adaptive control method according to any one of claims 1 to 8 is used to implement a sound adaptive control system, wherein the sound adaptive control system comprises: Mixed sound signal acquisition subsystem: It is composed of a single microphone, an array microphone and a matching professional sound signal acquisition and receiving device. It is used to collect effective sound signals and environmental noise data generated by the external environment, including background noise, human voice noise, etc. It is an important data source support subsystem of the present invention; Audio status detection and operation terminal: The audio status detection and operation terminal is located on a computer or server device, and includes a host computer software display interface and a hardware support module. The host computer software display interface is located on the computer device monitor, and can display the current internal mixed sound signal of the audio, the waveform and related parameters of each independent sound signal after blind source separation in real time, and provide the option of manual operation to control the audio; the hardware support module includes a camera module, an audio signal acquisition module, a central processing unit and a GPU computing power module. The camera module and the audio signal acquisition module are used to realize user authentication, voice control and gesture control. The central processing unit overall processes all data in the subsystem, and the GPU computing power module provides sufficient computing power support for neural network model training and deployment. This part is the core of the audio adaptive control system.

10. The adaptive sound control system according to claim 9, characterized in that: The adaptive sound control system further comprises: Noise adaptive filtering subsystem: This subsystem has a built-in adaptive filtering method for abnormal acoustic noise, including two modules: noise type recognition and noise adaptive filtering. The noise type recognition module has a built-in data preprocessing method and a neural network model, which can realize the data conversion from acoustic signals to acoustic signal waveform images. The neural network model can complete the identification of noise signal types and key parameters with the support of GPU computing power; the noise adaptive filtering module relies on adaptive variable bandpass filters and blind source separation technology to achieve noise separation and filtering; Adaptive control subsystem with visual and semantic dual recognition: This subsystem has a built-in adaptive control method based on visual and semantic dual recognition, and supports dual control methods of voice and gesture through two neural network models for performing semantic understanding and gesture recognition respectively.

Images