A mixing console-based audio data management system and method

Abstract

The application discloses a kind of audio data management system and method based on mixing console, it is related to mixing engineering management technical field, the present application collects audio, operation sequence, equipment state and spectral data, pre-processing is carried out;Through time domain, frequency domain and time-frequency analysis, audio and operation sequence features are extracted, long short-term memory network LSTM is used to capture time dynamics, long-distance dependence and interaction between features are captured by Transform;Association model is constructed by combining multi-modal fusion strategy, used to analyze the sound condition on site, determine instrument failure, normal off-site, microphone anomaly, audience seat noise and equipment switching abnormal situation, execute corresponding processing;In cloud environment, record the operation steps and timestamp of sound engineer, support multi-engineer collaborative processing, create independent version branch, realize version comparison, merging, evaluation and parameter difference visual analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sound engineering management, and particularly relates to an audio data management system and method based on a sound console. BACKGROUND

[0002] In large live performances or complex audio production environments, real-time management and sound processing of audio data are crucial to sound quality and live experience.

[0003] Traditional audio management relies on the experience of sound engineers and real-time monitoring, supplemented by basic hardware monitoring. However, this mode has significant shortcomings. For complex audio scenes, sound engineers are difficult to grasp the state changes of all sound sources in real time and comprehensively, and are prone to miss the key adjustment opportunity due to information overload or reaction delay, resulting in sound quality defects or equipment failure that cannot be handled in time. The existing technology often lacks deep correlation analysis capability for multi-source heterogeneous data such as audio data, operation sequence and device state, and is difficult to automatically identify and distinguish different types of abnormal situations, including instrument failure, microphone problem, audience noise or device switching, etc. Usually, sound engineers need to make judgments based on experience, which is inefficient and subjective, and the handling of abnormal situations is also prone to errors. In the cloud sound engineering scene of multi-person cooperation, although remote operation can be realized, there is a lack of fine-grained version management, collaborative processing mechanism and intuitive version comparison and evaluation tools, making it difficult for engineers to efficiently collaborate, and historical operation traceability and version comparison analysis are also difficult, which is not conducive to the management and quality control of complex projects. SUMMARY

[0004] The purpose of the present application is to provide an audio data management system and method based on a sound console to solve the problems in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides an audio data management method based on a sound console, comprising:

[0007] Collecting audio data, operation sequence data, device state and physical parameter data, and spectral energy distribution data, and preprocessing the collected data;

[0008] Through time domain, frequency domain and time-frequency analysis, the time domain, frequency domain and time-frequency features of the audio data, and the time sequence and state features of the operation sequence data are extracted; the long short-term memory network LSTM is used to capture the respective time dynamics, and the self-attention mechanism of the Transformer is used to capture the long-distance dependence and interaction between the features; combined with a multi-modal fusion strategy, the processed features are organically combined to construct an association model of the audio data and the operation sequence data;

[0009] The model analyzes the collected data to determine the sound condition and existing problems on site; when the audio signal, frequency band energy or device parameter of the musical instrument is abnormal, it is determined that the musical instrument has a fault or normally leaves the site; when the level, frequency band energy or time stamp offset of the human voice activity is abnormal, it is determined that the microphone is abnormal; when the noise level, noise spectrum or time matching condition of the audience area is abnormal, it is determined that the audience area has noise; when the audio signal, spectrum feature change or device operation log and time stamp matching are abnormal, it is determined that the device is switched; and corresponding processing is performed according to different conditions.

[0010] In the cloud tuning environment, the operations of the sound engineer are recorded as independent operation steps and versions, so that the engineer can collaboratively process the same audio and create and manage different mix versions, support version comparison, merging, evaluation and visual analysis of parameter differences.

[0011] In combination with the first aspect, in a first implementation manner of the first aspect of the present application, the collected audio data, operation sequence data, device state and physical parameter data and spectrum energy distribution data are preprocessed, including:

[0012] The audio data of the musical instrument, human voice and audience channel is collected through a high-precision multi-channel audio interface; the operation sequence data is collected through an internal logic interface of the mixing console; the device state and physical parameter data are collected through embedded sensors; and the real-time spectrum energy distribution data is collected through a spectrum analyzer to record the energy change of each frequency band and the spectrum dynamic characteristics.

[0013] The collected data is time-synchronized, a time stamp is added when the data is collected, and the data stream is aligned based on a unified time reference; a target format to be unified is determined, a programming language is used to convert all data into a unified format; and numerical data is normalized or standardized.

[0014] In combination with the first aspect, in a second implementation manner of the first aspect of the present application, the time domain, frequency domain and time-frequency characteristics of the audio data and the time sequence and state characteristics of the operation sequence data are extracted through time domain, frequency domain and time-frequency analysis; a long short-term memory network (LSTM) is used to capture the respective time dynamics, and a self-attention mechanism of a Transformer is used to capture long-distance dependence and interaction between features; a multi-modal fusion strategy is combined to organically combine the processed features, and an association model of the audio data and the operation sequence data is constructed, including:

[0015] The pre-processed audio data is subjected to time domain analysis, frequency domain analysis and time-frequency analysis; the time domain analysis divides the signal into short-time frames, calculates the root mean square energy and zero-crossing rate of each frame, and reveals the basic intensity characteristics and dynamic characteristics changing over time of the signal; the frequency domain analysis uses Fourier transform to convert the signal from time domain to frequency domain, identifies each frequency component and corresponding energy distribution constituting the signal; the time-frequency analysis divides the signal into time-varying windows and performs frequency domain analysis in each window to generate a time-frequency spectrum, which shows the time-varying characteristics of the frequency content of the signal; key features in these analysis results are extracted to form time domain feature vectors, frequency domain feature vectors and time-frequency feature vector sequences, which constitute the audio feature vector sequence;

[0016] The pre-processed operation sequence data is subjected to time sequence feature extraction and state feature extraction; each event or state point in the operation sequence is processed one by one in time sequence, based on the current time point, the past operations are reviewed, and the time sequence features are extracted; the values or states of each related component at the current time point are queried to extract the state features; all the time sequence features and state features extracted for the current time point are combined into a fixed-length numerical vector in a pre-defined order and format, and the feature vectors corresponding to each time point in the sequence are connected in time sequence to obtain a two-dimensional time sequence feature vector sequence;

[0017] The extracted audio feature vector sequence and the time sequence feature vector sequence of the operation sequence are respectively input into the LSTM network configured for each sequence, and the LSTM learns and outputs the dynamic change pattern representation of each sequence in the time dimension; the audio feature dynamic representation and the operation feature dynamic representation output by the LSTM are taken as input sequences and input into the Transformer model, and the self-attention mechanism of the Transformer is used to calculate the feature correlation weights within and between sequences, and the complex interaction and long-distance dependence between features are captured through the multi-head attention mechanism and the multi-layer perceptron;

[0018] A multi-modal fusion layer is designed to receive the time dynamic representation output by the LSTM and the interaction and dependence representation output by the Transformer; an attention mechanism is used to organically combine the feature representations from different models and capturing different information to generate a joint feature representation that can reflect the audio content, operation behavior and interaction at the same time; the joint feature representation is input into the final classifier to build the correlation model;

[0019] The correlation model is trained using labeled data, and the trained correlation model is deployed to receive real-time collected and pre-processed audio data and operation sequence data, dynamically update the internal state, and output real-time correlation analysis results, specifically the audio data, operation sequence data and the patterns of their correlation.

[0020] In a third implementation form of the first aspect, when the audio signal, the frequency band energy or the device parameter of the musical instrument is abnormal, it is determined that the musical instrument is malfunctioning or normally leaving, including:

[0021] The audio signal and the operation sequence of the musical instrument are monitored in real time, and are analyzed in combination with the association model; when the energy value of a specific frequency band is continuously lower than a preset silence threshold within a preset malfunction silence time, and there is no valid performance operation instruction during the period, and the association model predicts that this state is inconsistent with the current music context, it is determined that the musical instrument is malfunctioning; when the energy value of a specific frequency band or the harmonic structure exceeds a preset malfunction noise threshold within a preset malfunction noise time, and the change amplitude exceeds a preset malfunction allowable fluctuation range, and the association model fails to identify a pattern related to a performance skill or an environmental factor, it is determined that the musical instrument is malfunctioning; in order to exclude the case of poor contact of a device connection cable, device state data related to the musical instrument channel is checked, and the spectral characteristics of an abnormal period are analyzed; when the spectral characteristics are different from those of a malfunction of the musical instrument itself, it is determined that the problem is a line problem rather than a malfunction of the musical instrument.

[0022] When it is detected that the musical instrument is pressed or a mute instruction is received, and the signal energy value decreases to be lower than a preset silence threshold within a preset leaving decay time after the instruction is issued, and the association model confirms that the operation is consistent with a historical normal pattern, it is determined that the musical instrument is normally leaving; when the signal energy value is lower than a preset silence threshold within a preset leaving structure time, and the silence period matches a current music structure within a preset time tolerance threshold, and the association model fails to detect an abnormal pattern, it is determined that the musical instrument is normally leaving; when the signal energy value linearly decreases within a preset gradual change time, the rate is lower than a preset leaving allowable threshold, and finally decreases to be lower than a preset silence threshold, and the association model confirms that the process is consistent with a historical normal operation pattern, it is determined that the musical instrument is normally leaving; in order to exclude the case that a performer stops performing, but the musical instrument is still in an open state, and is only temporarily not used, device state data and operation sequence data are checked to see whether there is a power operation or an effecter switching, and the spectrum is analyzed; when there is a harmonic structure specific to the musical instrument, it is determined that the state is a stop-performing open temporarily unused state rather than a normal leaving.

[0023] In a fourth implementation form of the first aspect, when the level, the frequency band energy or the timestamp offset of the human voice activity of the human voice is abnormal, it is determined that the microphone is abnormal, including:

[0024] The collected human voice audio data is monitored in real time, and is analyzed in combination with the correlation model; the voice activity detection technology is used to identify the human voice area, and the human voice level, specific frequency band energy and human voice activity timestamp sequence of the area are extracted; the real-time extracted human voice level, frequency band energy and human voice activity timestamp offset are compared with the preset threshold; when the human voice level continuously exceeds the preset normal level range, and the duration exceeds the preset voice abnormal duration threshold, and the correlation model fails to identify the mode related to the intentional adjustment of the volume and the abnormal distance from the microphone of the singer, it is determined that the volume is abnormal; when the energy level of the non-human voice background noise continuously exceeds the preset noise threshold, and the duration exceeds the preset noise duration threshold, and the correlation model fails to identify the mode related to the breathing sound of the singer, the environmental noise or the singing skill, it is determined that the abnormal noise is abnormal; when the human voice activity timestamp sequence appears a long-time blank, and the duration exceeds the preset mute duration threshold, and the correlation model fails to identify the mode related to the normal pause, the improvisation processing or the normal mute of the singer, it is determined that the signal interruption is abnormal.

[0025] In combination with the first aspect, in a fifth implementation manner of the first aspect of the present application, when the noise level, noise spectrum or time matching condition of the audience area is abnormal, the audience noise is determined, including:

[0026] The audio signal from the microphone array of the audience area is monitored in real time, and is analyzed in combination with the correlation model; the real-time noise level, noise spectrum feature and noise activity time mode of the signal are extracted; the real-time extracted noise level, noise spectrum feature and noise activity time mode are compared with the preset threshold; when the noise level continuously exceeds the preset maximum allowed noise level threshold, the noise spectrum feature continuously deviates from the preset normal range threshold, or the noise activity time mode does not match the normal interaction mode threshold of the current performance stage, and the duration of the abnormal state exceeds the preset audience noise determination minimum duration threshold, and the correlation model determines that the noise mode does not match the normal interaction mode of the current performance stage, the audience noise is determined; in order to exclude the string sound of the instruments in other areas, the signals of other channels need to be compared; when similar sound is detected in other channels at the same time, and the original instrument spectrum feature is retained, it is determined that it is the string sound of the instruments in other areas rather than the audience noise.

[0027] In combination with the first aspect, in a sixth implementation manner of the first aspect of the present application, when the audio signal, spectrum feature change or device operation log and timestamp matching are abnormal, the device switching is determined, including:

[0028] Real-time monitoring of the audio signal of each channel of the mixing console, the spectral feature change and the operation log and timestamp of the related equipment, analyzing in combination with the correlation model; comparing the detected audio signal mutation amplitude, the rate and amplitude of the spectral feature change and the timestamp of the device operation log with the matching time difference of the audio signal mutation timestamp with the preset threshold; when the audio signal mutation amplitude exceeds the preset device switching signal mutation threshold, the rate or amplitude of the spectral feature change exceeds the preset device switching spectral change threshold, and the matching time difference of the timestamp of the device operation log and the audio signal mutation timestamp is within the preset device switching time matching tolerance threshold, or the device operation log records the switching instruction, and the duration of the abnormal state exceeds the preset device switching judgment minimum duration threshold, simultaneously the correlation model confirms that the detected state conforms to the feature mode of the device switching, the device switching is judged.

[0029] In combination with the first aspect, in a seventh implementation manner of the first aspect of the present application, the corresponding processing according to the different situations of the judgment includes:

[0030] When it is judged that the musical instrument is faulty, the time point of the fault occurrence is marked on the time axis of the audio data, the audio channel corresponding to the faulty musical instrument is marked, the volume of the channel is automatically reduced to mute, the technical personnel is notified, and the fault details are recorded in the log; when it is judged that the normal off-site occurs, the time point of the off-site occurrence is marked on the time axis of the audio data, the audio channel corresponding to the off-site musical instrument or singer is marked, the volume of the channel is controlled to be smoothly reduced according to a preset fade-out curve until mute, and the off-site information is recorded in the log;

[0031] When it is judged that the microphone is abnormal, the processing is performed according to the specific situation; when it is abnormal volume, the gain of the microphone channel is dynamically and smoothly adjusted to the preset normal level range through an automatic gain control algorithm in combination with the singing habit of the singer, the historical data of the song performance, the progress of the current song and the preset strategy; when it is abnormal noise, the time-domain audio signal collected by the microphone is converted to the frequency domain through short-time Fourier transform to obtain the amplitude spectrum and phase spectrum of each frame signal, the noise audio is subjected to short-time Fourier transform, and the average amplitude spectrum is calculated as the estimation of the noise amplitude spectrum; the estimated noise amplitude spectrum is subtracted from the signal amplitude spectrum using spectral subtraction, the pure speech amplitude spectrum obtained is combined with the original signal phase spectrum to form the processed spectrum, inverse short-time Fourier transform is performed, the time-domain audio signal is converted back, and the noise-reduced audio is obtained; when it is signal interruption, the microphone signal is tried to be automatically reconnected; if the reconnection is successful, the volume and equalization setting of the channel are restored according to the historical data and the song progress, if the reconnection fails, the channel is muted, the technical personnel is notified to check and immediately replace the equipment, and the details of the signal interruption are recorded in the log;

[0032] When it is judged that the audience noise is abnormal, the specific time range of the occurrence of the noise is marked on the time axis of the audio data, all audience audio input channels in the time period are marked as noise, the audience volume balance is automatically adjusted according to the noise type, the volume ratio of the audience in the main mix is reduced, and the type, duration and processing measures of the noise are recorded in the log;

[0033] When it is judged that the device is switched, the time point of the occurrence of the switching is marked on the time axis of the audio data, the audio channels involved in the switching are marked, the switching type is confirmed according to the device operation log, the parameters of the audio channels corresponding to the new device are automatically adjusted, the audio signal in the switching process is ensured to be continuous and free of mutation, and the detailed switching operation is recorded in the log.

[0034] In combination with the first aspect, in an eighth implementation manner of the first aspect of the application, in the cloud tuning environment, the operations of the sound engineers are recorded as independent operation steps and versions, so that the engineers can collaboratively process the same audio, create and manage respective mix versions, support version comparison, merging, evaluation and visual analysis of parameter differences, including:

[0035] The cloud server records and stores the operation steps and time stamps of the sound engineers, automatically generates mix versions, allows multiple engineers to collaboratively process and create independent version branches, supports version comparison, displays the audio waveform, spectrum graph difference and corresponding operation difference of different mix versions, supports version merging, allows engineers to selectively merge operations and parameters from different version branches, automatically processes conflicts and provides conflict resolution suggestions, supports listening evaluation of the generated mix versions, records the evaluation results and feedback, and provides graphical display of the parameter differences, displays the parameter changes between different versions in the form of charts or time axes, and facilitates engineers to intuitively analyze the differences between versions.

[0036] In a second aspect, the application provides an audio data management system based on a mixing console, including:

[0037] The data acquisition and preprocessing module includes a data acquisition unit and a data preprocessing unit; the data acquisition unit acquires audio data, operation sequence data, device state and physical parameter data and spectrum energy distribution data; the data preprocessing unit performs time synchronization on the acquired data, unifies the data format, and performs normalization or standardization processing on the numerical value type data;

[0038] The feature extraction and correlation model establishment module includes a feature extraction unit, a correlation model establishment unit and a model training unit; wherein the feature extraction unit performs multi-dimensional feature extraction on the pre-processed audio data and operation sequence data to form a feature vector sequence and a time sequence feature vector sequence; the correlation model establishment unit inputs the feature sequences of the audio and operation into LSTM to learn the time dynamic mode, and then inputs the LSTM output into the Transformer to capture the complex interaction and long distance dependence between features; a multi-modal fusion layer is designed to generate joint feature representation and build a correlation model; the model training unit trains the correlation model, receives real-time collected and pre-processed data, dynamically updates the state, and outputs real-time analysis results, audio data, operation sequence data and mutual correlation mode;

[0039] The abnormality detection and judgment module includes an abnormality detection unit and a situation judgment unit; wherein the abnormality detection unit uses the trained correlation model to monitor the audio data, operation sequence data, device state and physical parameter data and spectrum energy distribution data in real time, and identifies abnormal data; the situation judgment unit judges the abnormal data found by the abnormality detection unit, and divides them into instrument failure, normal departure, microphone abnormality, audience seat noise and device switching;

[0040] The abnormality processing module includes an abnormality processing unit and a log recording unit; wherein the abnormality processing unit executes the corresponding processing strategy according to the judgment result of the situation judgment unit; the log recording unit records the detailed information;

[0041] The cloud collaboration and version management module includes a version record and branch management unit, a version comparison and merging unit and an evaluation feedback and visualization unit; wherein the version record and branch management unit records the operation steps and time stamps of the sound engineer on the cloud server, generates a mix version, supports multi-engineer collaboration, creates independent version branches and works in parallel; the version comparison and merging unit displays the differences between different mix versions, supports version merging, allows engineers to selectively merge the operations of other branches, automatically handles conflicts and provides conflict resolution suggestions; the evaluation feedback and visualization unit supports evaluation of the generated mix version, records the evaluation results and feedback, and provides graphical display of parameter differences.

[0042] Compared with the prior art, the beneficial effects of the present application are:

[0043] 1、The present application fuses multi-modal data, extracts features using time domain, frequency domain and time-frequency analysis, adopts LSTM and Transformer deep learning models to capture the respective time dynamics and complex interaction and long distance dependence between features, and constructs an intelligent correlation model of audio and operation sequence.

[0044] 2、The application is based on the constructed correlation model, analyzes the collected data in real time, realizes the detection and judgment of the abnormality, and executes the corresponding processing strategy according to the judgment result.

[0045] 3、The application introduces the cloud end collaborative version management, and improves the collaboration efficiency and version control ability of the audio production. BRIEF DESCRIPTION OF DRAWINGS

[0046] Fig. 1 A step schematic diagram of an audio data management method based on a mixing console according to the application;

[0047] Fig. 2 A system structure diagram of an audio data management system based on a mixing console according to the application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the application will be apparently and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0049] Embodiment: As shown in the drawing, Figs. 1-2 the application provides a technical solution,

[0050] As shown in the drawing, Fig. 1 A step schematic diagram of an audio data management method based on a mixing console shows that the application provides an audio data management method based on a mixing console, which comprises the following steps:

[0051] Step S100: Collecting audio data, operation sequence data, device state and physical parameter data and spectrum energy distribution data, and pre-processing the collected data;

[0052] Specifically, the audio data of musical instruments, human voices and audience channels is collected through a high-precision multi-channel audio interface; the operation sequence data is collected through a mixing console internal logic interface; the device state and physical parameter data are collected through an embedded sensor; and the real-time spectrum energy distribution data is collected through a spectrum analyzer, recording the energy change of each frequency band and the spectrum dynamic characteristics.

[0053] The collected data is time-synchronized, a time stamp is added during data collection, and the data stream is aligned based on a unified time reference; a target format to be unified is determined, a programming language is used to convert all data into a unified format; and the numerical value type data is normalized or standardized.

[0054] In a specific embodiment, multi-track audio data is collected by a 24-channel Neumann U87 microphone array with a sampling rate of 48 kHz and a bit depth of 24 bits, recording audio streams of string instruments, brass instruments, vocals, and audience feedback, with peak levels of string instrument channels reaching -6 dBFS and audience channel background noise averaging -50 dBFS; operation sequence data is collected through the API interface of a Yamaha CL series mixing console, recording the change curve of the main sub from -12 dB smooth transition to -3 dB, real-time change data of reverb decay time from 2.5 s to 4.0 s, and timestamps of drum set solo scene activation; device status is collected by embedded sensors, monitoring the temperature of channel 1 microphone at 26.4°C, line current at 0.8A, and connection status showing as locked; spectral data is collected every 100 ms by a spectrum analyzer, recording dynamic changes of 1 kHz band energy at -20 dB and 10 kHz band energy at -35 dB.

[0055] In the preprocessing stage, the collected 24-track audio is time-synchronized using the PyAudioAnalysis library of Python, the timestamp error is calibrated to within ±1 ms through the NTP protocol, all data is converted to a unified structure based on JSON format, and numerical data is processed by Min-Max normalization to map values to the [0,1] interval.

[0056] Step S200: Through time domain, frequency domain and time-frequency analysis, the time domain, frequency domain and time-frequency features of the audio data, and the time sequence and state features of the operation sequence data are extracted; the long short-term memory network LSTM is used to capture the respective time dynamics, and the self-attention mechanism of the Transformer is used to capture the long-distance dependence and interaction between features; combined with the multi-modal fusion strategy, the processed features are organically combined to construct the association model of audio data and operation sequence data;

[0057] Specifically, the preprocessed audio data is subjected to time domain analysis, frequency domain analysis and time-frequency analysis; time domain analysis divides the signal into short-time frames, calculates the root mean square energy and zero-crossing rate of each frame, and reveals the basic intensity characteristics and dynamic characteristics over time; frequency domain analysis uses Fourier transform to convert the signal from time domain to frequency domain, identifies the frequency components and corresponding energy distribution that make up the signal; time-frequency analysis divides the signal into time-varying windows and performs frequency domain analysis within each window to generate a time-frequency spectrogram, showing the time-varying characteristics of the signal frequency content; key features are extracted from these analysis results to form time domain feature vectors, frequency domain feature vectors and time-frequency feature vector sequences, constituting the audio feature vector sequence;

[0058] The time sequence features and state features are extracted from the preprocessed operation sequence data; each event or state point in the operation sequence is processed in time sequence, the time sequence features are extracted based on the current time point and the past operations, the state features are extracted by querying the values or states of each related component at the current time point, all the time sequence features and state features extracted for the current time point are combined into a fixed-length numerical vector in a predefined order and format, the feature vectors corresponding to each time point in the sequence are connected in time sequence to obtain a two-dimensional time sequence feature vector sequence;

[0059] The extracted audio feature vector sequence and the time sequence feature vector sequence of the operation sequence are respectively input into the respective configured LSTM network, the LSTM learns and outputs the dynamic change mode representation of each sequence in the time dimension; the audio feature dynamic representation and the operation feature dynamic representation output by the LSTM are taken as input sequences and input into the Transformer model, the feature correlation weights between the sequences and within the sequences are calculated by using the self-attention mechanism of the Transformer, the dynamic representations of the audio and operation sequences are projected into different feature spaces through linear transformation, position encoding is added to introduce sequence order information, for each position in the sequence, the correlation scores between the position and all positions in the sequence and all positions in the other sequence are calculated, the scaled dot-product attention is used to measure, which is obtained through the interaction of query, key and value matrices, after the scores are normalized by softmax, the feature correlation weights are obtained, which represent the importance of each position to the current position, the multi-head attention mechanism further parallelizes this process, captures different types of correlations through multiple independent attention heads, splices and linearly projects the outputs of each head to obtain a feature representation that integrates the complex dependency relationships between the sequences and within the sequences;

[0060] A multi-modal fusion layer is designed to receive the time dynamic representation output by the LSTM and the interaction and dependency representation output by the Transformer; an attention mechanism is used to organically combine the feature representations from different models and capturing different information to generate a joint feature representation that can reflect the audio content, operation behavior and interaction at the same time; the joint feature representation is input into the final classifier to build a correlation model;

[0061] The multi-modal data segments containing audio data and corresponding operation sequences are collected, ensuring that both are accurately synchronized in time, a clear predefined label is assigned to each data segment, describing the audio content, operation behavior and its associated pattern; through data alignment, ensure that the audio features and operation features of each time step strictly correspond to the label, and divide the training set, validation set and test set, use the training set and validation set to train the association model, optimize the model parameters through forward propagation, loss calculation, back propagation and parameter update, while monitoring the performance on the validation set and adjusting the hyperparameters, after training is completed, the test set is used to evaluate the final model, select the model version with the best performance and strong generalization ability for saving, deploy the trained association model, receive real-time collected and preprocessed audio data and operation sequence data, dynamically update the internal state, and output the correlation analysis results in real time, specifically the audio data, operation sequence data and the pattern associated between them.

[0062] In a specific embodiment, the preprocessed audio data is feature extracted, the 48kHz audio signal of the string channel is segmented into 20ms short-time frames, the root mean square energy and zero-crossing rate of each frame are calculated to form a time-domain feature vector sequence; 1024-point fast Fourier transform is used for frequency domain analysis, the main frequency component is identified at 440Hz, energy-10dB and 880Hz, energy-15dB, a frequency domain feature vector sequence is generated; time-frequency analysis is performed through short-time Fourier transform, a 128x128 time-frequency spectrogram is generated, the 440Hz frequency component is captured to appear obvious energy enhancement in the 200-250th frame, from-15dB to-5dB, a time-frequency feature vector sequence is formed, and finally an audio feature vector sequence containing 3000 time-domain feature vectors, frequency domain feature vectors and time-frequency feature vectors is formed; the operation sequence data is extracted for timing feature, the average frequency of the pusher position adjustment operation is calculated as 0.5 times per second, the average interval time of two operations is 2.1 seconds, the state feature is extracted, the current pusher position is recorded as-3.5dB, the knob angle is 72 degrees and the activated effect unit is the reverb unit, an operation sequence feature vector sequence containing 1500 timing feature vectors and state feature vectors is formed.

[0063] The audio feature vector sequence is input into an LSTM network configured with 128 hidden units, and the operation sequence feature vector sequence is input into an LSTM network configured with 64 hidden units. The LSTMs output 3000 and 1500 dynamic change mode representations, respectively. The audio dynamic representation and the operation dynamic representation output by the LSTMs are input as an input sequence into a Transformer model. Eight attention heads are set, and the feature correlation weights are calculated using a self-attention mechanism. A multi-modal fusion layer is designed to fuse the time dynamic representation output by the LSTMs and the interaction representation output by the Transformer using a gating mechanism to generate a joint feature representation containing 4096 dimensions. The joint feature representation is input into a final 3-layer fully connected classifier to build a correlation model.

[0064] The correlation model is trained using a labeled dataset containing 1000 samples. After training, the model is deployed to receive and analyze current audio data and operation sequence data in real time, dynamically update internal states, and output real-time correlation analysis results.

[0065] Step S300: The model analyzes the collected data to determine the sound conditions and existing problems on site. When the audio signal, frequency band energy, or device parameter of the musical instrument is abnormal, it is determined that a musical instrument fault or normal off-site has occurred. When the level, frequency band energy, or timestamp offset of the human voice activity is abnormal, it is determined that the microphone is abnormal. When the noise level, noise spectrum, or time matching condition in the audience area is abnormal, it is determined that the audience noise is abnormal. When the audio signal, frequency spectrum feature change, or device operation log and timestamp matching are abnormal, it is determined that the device switching is abnormal.

[0066] Specifically, the audio signal and operation sequence of the musical instrument are monitored in real time, and the correlation model is analyzed. When the energy value of a specific frequency band is continuously lower than the preset silence threshold within the preset fault silence time, and there is no valid performance operation instruction during the period, and the correlation model predicts that this state is inconsistent with the current music context, it is determined that the musical instrument is faulty. When the energy value or harmonic structure of a specific frequency band of the signal exceeds the preset fault noise threshold within the preset fault noise time, and the change amplitude exceeds the preset fault allowable fluctuation range, and the correlation model fails to identify patterns related to performance skills or environmental factors, it is determined that the musical instrument is faulty. In order to exclude the case of poor contact of the device connection cable, the device state data related to the instrument channel is checked, and the spectrum features of the abnormal period are analyzed. When the spectrum features are different from those of the instrument itself, it is determined that the problem is the line rather than the instrument.

[0067] When the instrument is detected to be pressed or a mute instruction is received, and the signal energy value decreases to below the preset silence threshold within the preset off-site decay time after the instruction is issued, and the associated model confirms that the operation is consistent with the historical normal mode, it is judged as normal off-site; when the signal energy value is below the preset silence threshold within the preset off-site structure time, and the silence period matches the current music structure within the preset time tolerance threshold, and the associated model does not detect abnormal mode, it is judged as normal off-site; when the signal energy value linearly decreases within the preset gradual change time, the rate is lower than the preset off-site allowed threshold, and finally below the preset silence threshold, and the associated model confirms that the process is consistent with the historical normal operation mode, it is judged as normal off-site; in order to exclude the case that the performer stops playing, but the instrument is still in the open state, only temporarily not used, the device state data and operation sequence data are checked to see if there is a power operation or effecter switching, the spectrum is analyzed, and when there is a harmonic structure unique to the instrument, it is judged as a stop playing state of opening and not being used, rather than a normal off-site.

[0068] Real-time monitoring is performed on the collected human voice audio data, and analysis is performed in combination with the associated model; voice activity detection technology is used to identify the human voice area, extract the human voice level, specific frequency band energy and timestamp sequence of human voice activity in the area; the real-time extracted human voice level, frequency band energy and timestamp offset of human voice activity are compared with the preset threshold; when the human voice level continuously exceeds the preset normal level range, and the duration exceeds the preset voice abnormal duration threshold, and the associated model fails to identify the mode related to the singer's intentional adjustment of volume and distance from the microphone, it is judged as volume abnormality; when the energy level of non-human voice background noise continuously exceeds the preset noise threshold, and the duration exceeds the preset noise duration threshold, and the associated model fails to identify the mode related to the singer's breathing sound, environmental noise or singing skill, it is judged as abnormal noise; when the timestamp sequence of human voice activity appears long time blank, and the duration exceeds the preset silence duration threshold, and the associated model fails to identify the mode related to the singer's normal pause, improvisation or normal silence, it is judged as signal interruption.

[0069] Real-time monitoring of audio signals from the audience area microphone array, combined with correlation model analysis; extract the real-time noise level, noise spectrum features and noise activity time pattern of the signal; compare the real-time extracted noise level, noise spectrum features and noise activity time pattern with the preset threshold; when the noise level continuously exceeds the preset maximum allowed noise level threshold, the noise spectrum feature continuously deviates from the preset normal range threshold, or the noise activity time pattern does not match the normal interaction mode threshold of the current performance stage, and the abnormal state duration exceeds the preset audience seat noise judgment minimum duration threshold, and the correlation model judges that the noise mode does not match the normal interaction mode of the current performance stage, it is judged as audience seat noise; in order to exclude the sound of other area instruments, it is necessary to compare the signals of other channels, when similar sound is detected in other channels, and the original instrument spectrum feature is retained, it is judged as other area instrument sound instead of audience seat noise.

[0070] Real-time monitoring of audio signals, spectrum feature changes and related device operation logs and timestamps of each channel of the mixing console, combined with correlation model analysis; compare the detected audio signal mutation amplitude, spectrum feature change rate and amplitude, and device operation log timestamp with the matching time difference of the audio signal mutation timestamp with the preset threshold; when the audio signal mutation amplitude exceeds the preset device switching signal mutation threshold, the spectrum feature change rate or amplitude exceeds the preset device switching spectrum change threshold, and the matching time difference of the device operation log timestamp and the audio signal mutation timestamp is within the preset device switching time matching tolerance threshold, or the device operation log records the switching instruction, and the abnormal state duration exceeds the preset device switching judgment minimum duration threshold, and the correlation model confirms that the detected state meets the characteristics of device switching, it is judged as device switching.

[0071] In a specific embodiment, the model analyzes the collected data in real time to determine the sound condition on site; the audio signal of the violin channel is monitored for 10 seconds of fault silence time from 15:23:00 to 15:23:10, the energy value of the 1000Hz frequency band is continuously lower than the silence threshold of-60dBFS, there is no violin playing operation instruction in the operation sequence during the period, and the correlation model predicts that this silence state does not match the music piece that should be continuously played at present, and it is judged as violin failure.

[0072] Real-time monitoring of the human voice channel, using speech activity detection technology to identify the human voice area, extract the human voice activity timestamp sequence from 15:27:10 to 15:27:30, the area human voice level is between-15dBFS to-10dBFS, which exceeds the preset normal level range-20dBFS to-12dBFS, the duration exceeds 3 seconds of sound abnormal duration threshold, and the associated model fails to identify the pattern related to the deliberate adjustment of the singer, judging as volume abnormality.

[0073] Real-time monitoring of the audience area microphone array, extracting the noise level from 15:30:00 to 15:30:45, which continuously exceeds the maximum allowed noise level threshold of-25dBFS, the noise spectrum feature continuously deviates from the normal range above 5000Hz frequency band, the noise activity time pattern does not match the normal interaction pattern of the current performance stage, the audience noise determination minimum duration threshold exceeds 10 seconds of abnormal state duration, and the associated model judges that the noise pattern does not match the normal interaction pattern, judging as audience noise.

[0074] Step S400: according to the different situations of judgment, corresponding processing is carried out;

[0075] Specifically, when it is judged that the instrument is faulty, the time point of the fault occurrence is marked on the time axis of the audio data, the audio channel corresponding to the faulty instrument is marked, the volume of the channel is automatically reduced to mute, the technical personnel is notified, and the fault details are recorded in the log; when it is judged that it is normal to leave, the time point of the leaving occurrence is marked on the time axis of the audio data, the audio channel corresponding to the leaving instrument or singer is marked, the volume of the channel is controlled to be smoothly reduced according to the preset fade-out curve until mute, and the leaving information is recorded in the log;

[0076] When it is judged that the microphone is abnormal, processing is performed according to specific circumstances; when it is volume abnormal, the singing habits of the singer, historical data of song performance, progress of the current song and preset strategies are combined, the gain of the microphone channel is dynamically and smoothly adjusted through an automatic gain control algorithm to restore to a preset normal level range; when it is abnormal noise, the time domain audio signal collected by the microphone is converted to the frequency domain through short-time Fourier transform to obtain the amplitude spectrum and phase spectrum of each frame signal, the average amplitude spectrum is calculated as the estimation of the noise amplitude spectrum through short-time Fourier transform of the noise audio, the estimated noise amplitude spectrum is subtracted from the signal amplitude spectrum using spectral subtraction, the obtained pure speech amplitude spectrum is combined with the original signal phase spectrum to form the processed spectrum, inverse short-time Fourier transform is performed to convert back to the time domain audio signal to obtain the noise-reduced audio; when it is signal interruption, the microphone signal is automatically reconnected; if the reconnection is successful, the volume and equalization settings of the channel are restored according to the historical data and the progress of the song, if the reconnection fails, the channel is muted, technicians are notified to check and replace the equipment immediately, and the details of the signal interruption are recorded in the log;

[0077] When it is judged that the audience seat noise is abnormal, the specific time range of the occurrence of the noise abnormality is marked on the time axis of the audio data, all audience seat audio input channels in this time period are marked as noise abnormality, the audience seat volume balance is automatically adjusted according to the noise type, the volume proportion of the audience seat in the main mix is reduced, and the type, duration and processing measures of the noise abnormality are recorded in the log;

[0078] When it is judged that the device is switched, the time point of the switching is marked on the time axis of the audio data, the audio channels involved in the switching are marked, the switching type is confirmed according to the device operation log, the parameters of the audio channels corresponding to the new device are automatically adjusted to ensure that the audio signal during the switching process is continuous and has no mutation, and detailed switching operations are recorded in the log.

[0079] In a specific embodiment, it is judged that the violin channel has a musical instrument failure at 15:23:00, the time point is immediately marked on the time axis of the audio data, the violin channel is marked as failure, its volume is automatically reduced to mute, and a notification "Channel 1 violin failure, please check" is sent to the background, and "15:23:00 Channel 1 violin failure, mute processing" is recorded in the log.

[0080] It is judged that the vocal channel has abnormal noise from 15:27:10 to 15:27:30, the average amplitude spectrum of the noise audio is calculated through STFT conversion of the collected audio signal, the noise amplitude spectrum is subtracted from the original signal amplitude spectrum using spectral subtraction to obtain the pure speech amplitude spectrum, and the noise-reduced audio is obtained after inverse STFT of the combined pure speech amplitude spectrum and original phase spectrum, and the log records "15:27:10-15:27:30 Channel 5 vocal noise reduction processing".

[0081] Determine that a device switch occurs at 15:31:20, mark this point on the timeline, mark the master output channel as a device switch, confirm that it is a scene switch according to the operation log, automatically adjust the volume, sound image and effect parameters of each channel in the new scene to ensure a smooth switching process, and log "15:31:20 Scene switch, channel parameter automatic adjustment".

[0082] Step S500: In a cloud mixing environment, record the mixing engineer's operations as independent operation steps and versions, allowing engineers to collaboratively process the same audio, create and manage their own mix versions, support version comparison, merging, evaluation and visual analysis of parameter differences.

[0083] Specifically, the cloud server records and stores the mixing engineer's operation steps and timestamps, automatically generates mix versions, allows multiple engineers to collaboratively process and create independent version branches, supports version comparison to display audio waveform, spectrum graph differences and corresponding operation differences between different mix versions, supports version merging to allow engineers to selectively merge operations and parameters from different version branches, automatically handle conflicts and provide conflict resolution suggestions, supports perceptual evaluation of generated mix versions, records evaluation results and feedback, and provides graphical display of parameter differences in the form of charts or timelines to visually analyze the differences between versions.

[0084] In a specific embodiment, in a cloud mixing environment, mixing engineer A is processing song X, records his operation steps, adjusts the master output fader from -3dB to -1dB at 15:40:05, and adds a compressor to channel 2 bass at 15:40:20 with a threshold of -18dB and a ratio of 4:1. Automatically generate version v1.0, store the operation and timestamp. Engineer B creates an independent branch v1.1 from v1.0, adjusts the EQ of channel 1 voice at 15:45:10, and adjusts the 1kHz gain from 0dB to +3dB. The version comparison function shows that v1.0 and v1.1 differ in EQ parameters of channel 1 and effects of channel 2, and visually displays the subtle changes in waveform and spectrum. Engineer A decides to merge the branches, chooses to keep B's EQ adjustment and discard B's compressor addition, and the system automatically handles the conflict in the master output fader adjustment. After merging, version v1.2 is generated. Engineer A uses the perceptual evaluation function to score v1.2 8 / 10 and adds a note "voice is clearer, low frequency is slightly insufficient". The parameter difference visualization chart shows that in versions v1.0 to v1.2, the 1kHz parameter of channel 1 changes from 0dB to +3dB at 15:45:10, and the compressor parameter of channel 2 is added at 15:40:20.

[0085] As Fig. 2A system structure diagram of a mixing console-based audio data management system is shown, and the present application provides a mixing console-based audio data management system, comprising:

[0086] A data acquisition and preprocessing module includes a data acquisition unit and a data preprocessing unit; wherein the data acquisition unit acquires audio data, operation sequence data, device state and physical parameter data, and spectral energy distribution data; the data preprocessing unit performs time synchronization on the collected data, unifies the data format, and normalizes or standardizes numerical data;

[0087] A feature extraction and correlation model establishment module includes a feature extraction unit, a correlation model establishment unit, and a model training unit; wherein the feature extraction unit performs multi-dimensional feature extraction on the preprocessed audio data and operation sequence data to form feature vector sequences and time sequence feature vector sequences; the correlation model establishment unit inputs the feature sequences of audio and operation into LSTM to learn time dynamic patterns, and then inputs the LSTM output into Transformer to capture complex interactions and long-distance dependencies between features; a multi-modal fusion layer is designed to generate joint feature representation and build a correlation model; the model training unit trains the correlation model, receives real-time collected and preprocessed data, dynamically updates the state, and outputs real-time analysis results, including audio data, operation sequence data, and interrelated patterns;

[0088] An anomaly detection and judgment module includes an anomaly detection unit and a situation judgment unit; wherein the anomaly detection unit uses the trained correlation model to monitor audio data, operation sequence data, device state and physical parameter data, and spectral energy distribution data in real time, and identifies abnormal data; the situation judgment unit judges the abnormal data found by the anomaly detection unit, and divides them into instrument failure, normal departure, microphone anomaly, audience seat noise, and device switching;

[0089] An anomaly handling module includes an anomaly handling unit and a log recording unit; wherein the anomaly handling unit executes corresponding handling strategies according to the judgment results of the situation judgment unit; the log recording unit records detailed information;

[0090] A cloud collaboration and version management module includes a version record and branch management unit, a version comparison and merging unit, and an evaluation feedback and visualization unit; wherein the version record and branch management unit records the operation steps and timestamps of the sound engineer on the cloud server, generates a mix version, supports multi-engineer collaboration, creates independent version branches for parallel work; the version comparison and merging unit displays the differences between different mix versions, supports version merging, allows engineers to selectively merge operations from other branches, automatically handles conflicts and provides conflict resolution suggestions; the evaluation feedback and visualization unit supports evaluation of the generated mix version, records evaluation results and feedback, and provides graphical display of parameter differences.

[0091] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The presently disclosed embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference herein to any prior art is to be taken as an admission that the present application is not entitled to antedate such prior art by virtue of prior application. No reference to an item of prior art in any claim is to be construed as an admission that the present application is not entitled to priority to such prior art by virtue of prior application. No reference in the specification to an item of prior art is to be construed as an admission that the present application is not entitled to priority to such prior art by virtue of prior application. No reference in the specification to an item of prior art is to be construed as an admission that the present application is not entitled to priority to such prior art by virtue of prior application. No reference in the specification to an item of prior art is to be construed as an admission that the present application is not entitled to priority to such prior art by virtue of prior application. No reference in the specification to an item of prior art is to be construed as an admission

Claims

1. A method for managing audio data based on a sound console, characterized by, The method comprises the following steps: Collecting audio data, operation sequence data, device state and physical parameter data, and spectral energy distribution data, and preprocessing the collected data; Through time domain, frequency domain and time-frequency analysis, the time domain, frequency domain and time-frequency features of the audio data, and the time sequence and state features of the operation sequence data are extracted; the long short-term memory network (LSTM) is used to capture the time dynamics of each feature, and the self-attention mechanism of the Transformer is used to capture the long-distance dependence and interaction between features; a multi-modal fusion strategy is combined to organically combine the processed features, and an association model of the audio data and the operation sequence data is constructed; Through time domain, frequency domain and time-frequency analysis, the time domain, frequency domain and time-frequency features of the audio data, and the time sequence and state features of the operation sequence data are extracted; the long short-term memory network (LSTM) is used to capture the time dynamics of each feature, and the self-attention mechanism of the Transformer is used to capture the long-distance dependence and interaction between features; A multi-modal fusion strategy is combined to organically combine the processed features, and an association model of the audio data and the operation sequence data is constructed, comprising: The preprocessed audio data is subjected to time domain analysis, frequency domain analysis and time-frequency analysis; the time domain analysis divides the signal into short-time frames, calculates the root mean square energy and zero-crossing rate of each frame, and reveals the basic intensity features and dynamic characteristics of the signal over time; the frequency domain analysis uses Fourier transform to convert the signal from the time domain to the frequency domain, and identifies the frequency components and corresponding energy distribution of the signal; the time-frequency analysis divides the signal into time-varying windows and performs frequency domain analysis in each window to generate a time-frequency spectrum, which shows the time-varying characteristics of the signal frequency content; key features are extracted from these analysis results to form time domain feature vectors, frequency domain feature vectors and time-frequency feature vector sequences, which constitute the audio feature vector sequence; The preprocessed operation sequence data is subjected to time sequence feature extraction and state feature extraction; each event or state point in the operation sequence is processed in time sequence, and based on the current time point, the past operations are reviewed to extract the time sequence features; the values or states of each related component at the current time point are queried to extract the state features; all the time sequence features and state features extracted for the current time point are combined into a fixed-length numerical vector in a pre-defined order and format, and the feature vectors corresponding to each time point in the sequence are connected in time sequence to obtain a two-dimensional time sequence feature vector sequence; The obtained audio feature vector sequence and the time sequence feature vector sequence of the operation sequence are respectively input into the respective configured LSTM network, and the LSTM learns and outputs the dynamic change mode representation of each sequence in the time dimension; the audio feature dynamic representation and the operation feature dynamic representation output by the LSTM are taken as input sequences and input into the Transformer model, and the self-attention mechanism of the Transformer is used to calculate the feature correlation weights between the sequences and between the sequences; through the multi-head attention mechanism and the multi-layer perceptron, the complex interaction relationship and long-distance dependence between the features are captured. A multi-modal fusion layer is designed to receive the time dynamic representation of the LSTM output and the interaction and dependency representation of the Transformer output; an attention mechanism is used to organically combine the feature representations from different models and capturing different information to generate a joint feature representation that can reflect the audio content, operation behavior and interaction at the same time; the joint feature representation is input into the final classifier to build the correlation model; The correlation model is trained using labeled data, and the trained correlation model is deployed to receive real-time collected and pre-processed audio data and operation sequence data, dynamically update the internal state, and output real-time correlation analysis results, specifically audio data, operation sequence data and the mode of their correlation; The correlation model analyzes the collected data to determine the sound condition and existing problems on site; when the audio signal, frequency band energy or device parameter of the musical instrument is abnormal, it is determined that the musical instrument has a fault or normally leaves; when the level, frequency band energy or timestamp offset of the human voice activity is abnormal, it is determined that the microphone is abnormal; when the noise level, noise spectrum or time matching condition of the audience area is abnormal, it is determined that the audience area is noisy; when the audio signal, spectrum feature change or device operation log and timestamp matching are abnormal, it is determined that the device is switched; and corresponding processing is performed according to different situations; In a cloud tuning environment, the operations of the sound engineer are recorded as independent operation steps and versions, allowing engineers to collaboratively process the same audio and create and manage their own mix versions, supporting version comparison, merging, evaluation and visual analysis of parameter differences.

2. The audio data management method based on a mixing console according to claim 1, characterized in that, The collected audio data, operation sequence data, device state and physical parameter data, and spectrum energy distribution data are pre-processed, including: Audio data of musical instruments, human voices and audience channels are collected through high-precision multi-channel audio interfaces; operation sequence data is collected through internal logic interfaces of the mixing console; device state and physical parameter data are collected through embedded sensors; real-time spectrum energy distribution data is collected through spectrum analyzers to record the energy changes of each frequency band and the dynamic characteristics of the spectrum; The collected data are time-synchronized, and timestamps are added during data collection to ensure that the data stream is aligned based on a unified time reference; the target format to be unified is determined, and all data are converted to a unified format using a programming language; numerical data are normalized or standardized.

3. The audio data management method based on a mixing console according to claim 1, characterized in that, When the audio signal, frequency band energy or device parameter of the musical instrument is abnormal, it is determined that the musical instrument has a fault or normally leaves, including: Real-time monitoring of the audio signal and operation sequence of the musical instrument, combined with correlation model analysis; when the energy value of a specific frequency band is continuously lower than the preset silent threshold within the preset fault silent time, and there is no effective performance operation instruction during the period, and the correlation model predicts that this state is inconsistent with the current music context, it is judged as a musical instrument fault; when the energy value or harmonic structure of a specific frequency band of the signal exceeds the preset fault noise threshold within the preset fault noise time, and the change amplitude exceeds the preset fault allowable fluctuation range, and the correlation model fails to identify the mode related to the performance skill or environmental factors, it is judged as a musical instrument fault; in order to exclude the case of poor contact of the device connection cable, the device state data related to the instrument channel is checked, the spectral characteristics of the abnormal period are analyzed, and when the spectral characteristics are different from those of the instrument itself fault, it is judged as a line problem rather than an instrument fault; When it is detected that the musical instrument is pressed or receives a mute instruction, and the signal energy value decreases to below the preset silent threshold within the preset off-site decay time after the instruction is issued, and the correlation model confirms that the operation is consistent with the historical normal mode, it is judged as normal off-site; when the signal energy value is lower than the preset silent threshold within the preset off-site structure time, and the silent period matches the current music structure within the preset time tolerance threshold, and the correlation model does not detect abnormal mode, it is judged as normal off-site; when the signal energy value linearly decreases within the preset gradual change time, the rate is lower than the preset off-site allowable threshold, and finally lower than the preset silent threshold, and the correlation model confirms that the process is consistent with the historical normal operation mode, it is judged as normal off-site; in order to exclude the case that the performer stops playing, but the musical instrument is still in the open state, only temporarily not used, the device state data and operation sequence data are checked to see if there is power operation or effecter switching, the spectrum is analyzed, and when there is a harmonic structure unique to the musical instrument, it is judged as a stop playing open temporarily not used state rather than a normal off-site.

4. The audio data management method based on a mixing console according to claim 1, characterized in that, The judgment of the microphone abnormality when the level, frequency band energy or time stamp offset of the human voice is abnormal includes: Real-time monitoring of collected human voice audio data, combined with correlation model analysis; using voice activity detection technology to identify human voice area, extracting human voice level, specific frequency band energy and human voice activity timestamp sequence in the area; comparing the real-time extracted human voice level, frequency band energy and human voice activity timestamp offset with the preset threshold; when the human voice level continuously exceeds the preset normal level range, and the duration exceeds the preset voice abnormal duration threshold, and the correlation model fails to identify the pattern related to the singer's deliberate adjustment of volume and abnormal distance from the microphone, it is judged as volume abnormality; when the energy level of non-human voice background noise continuously exceeds the preset noise threshold, and the duration exceeds the preset noise duration threshold, and the correlation model fails to identify the pattern related to the singer's breathing sound, environmental noise or singing skill, it is judged as abnormal noise; When the human voice activity timestamp sequence appears long-time blank, and the duration exceeds the preset mute duration threshold, and the correlation model fails to identify the pattern related to the singer's normal pause, improvisation or normal mute, it is judged as signal interruption.

5. The audio data management method based on a mixing console according to claim 1, characterized in that, When the noise level, noise spectrum or time matching condition of the audience area is abnormal, it is judged as audience noise, including: Real-time monitoring of audio signals from the audience area microphone array, combined with correlation model analysis; extracting real-time noise level, noise spectrum features and noise activity time patterns of the signals; comparing the real-time extracted noise level, noise spectrum features and noise activity time patterns with the preset threshold; when the noise level continuously exceeds the preset maximum allowed noise level threshold, the noise spectrum features continuously deviate from the preset normal range threshold, or the noise activity time pattern does not match the normal interaction mode threshold of the current performance stage, and the duration of the abnormal state exceeds the preset audience noise judgment minimum duration threshold, and the correlation model judges that the noise pattern does not match the normal interaction mode of the current performance stage, it is judged as audience noise; In order to exclude the sound from other area instruments, the signals of other channels need to be compared, when similar sound is detected in other channels at the same time, and the original instrument spectrum features are retained, it is judged as other area instrument sound instead of audience noise.

6. The audio data management method based on a mixing console according to claim 1, characterized in that, When the audio signal, spectrum feature change or device operation log and timestamp matching appear abnormal, it is judged as device switching, including: Real-time monitoring of the audio signal of each channel of the mixing console, the spectral feature change and the operation log and timestamp of the related equipment, combined with the correlation model for analysis; the detected audio signal mutation amplitude, the rate and amplitude of the spectral feature change and the timestamp of the equipment operation log are compared with the matching time difference of the audio signal mutation timestamp and the preset threshold; when the audio signal mutation amplitude exceeds the preset device switching signal mutation threshold, the spectral feature change rate or amplitude exceeds the preset device switching spectral change threshold, and the matching time difference of the timestamp of the equipment operation log and the audio signal mutation timestamp is within the preset device switching time matching tolerance threshold, or the equipment operation log records the switching instruction, and the duration of the abnormal state exceeds the preset device switching judgment minimum duration threshold, while the correlation model confirms that the detected state meets the feature mode of device switching, it is judged as device switching.

7. The audio data management method based on a mixing console according to claim 1, characterized in that, The corresponding processing is carried out according to the different situations, including: When it is judged that the instrument is faulty, the time point of the fault occurrence is marked on the time axis of the audio data, the audio channel corresponding to the faulty instrument is marked, the volume of the channel is automatically reduced to mute, the technician is notified, and the fault details are recorded in the log; when it is judged that the normal off-site, the time point of the off-site occurrence is marked on the time axis of the audio data, the audio channel corresponding to the off-site instrument or singer is marked, the volume of the channel is controlled to be smoothly reduced according to the preset fade-out curve until mute, and the off-site information is recorded in the log; When it is judged that the microphone is abnormal, the specific situation is handled; when the volume is abnormal, the singer's singing habit, historical data of song performance, current song progress and preset strategy are combined, the gain of the microphone channel is dynamically and smoothly adjusted through the automatic gain control algorithm, and restored to the preset normal level range; when it is abnormal noise, the time-domain audio signal collected by the microphone is converted to the frequency domain through short-time Fourier transform to obtain the amplitude spectrum and phase spectrum of each frame signal, the noise amplitude spectrum is estimated by calculating the average amplitude spectrum of the noise audio signal through short-time Fourier transform; the estimated noise amplitude spectrum is subtracted from the signal amplitude spectrum using spectral subtraction, the pure speech amplitude spectrum obtained is combined with the original signal phase spectrum to form the processed spectrum, and inverse short-time Fourier transform is performed to convert the time-domain audio signal to obtain the noise-reduced audio; when it is signal interruption, try to automatically reconnect the microphone signal; if the reconnection is successful, the volume and equalization setting of the channel are restored according to the historical data and the song progress, and if the reconnection fails, the channel is muted, the technician is notified to check and immediately replace the equipment, and the details of the signal interruption are recorded in the log; When it is judged that the audience seat noise is abnormal, the specific time range of the noise abnormality occurrence is marked on the time axis of the audio data, all audience seat audio input channels in this time period are marked as noise abnormality, the audience seat volume balance is automatically adjusted according to the noise type, the volume proportion of the audience seat in the main mix is reduced, and the type, duration and processing measures of the noise abnormality are recorded in the log. When it is judged that the device is switched, the time point when the switching occurs is marked on the time axis of the audio data, the audio channel involved in the switching is marked, the switching type is confirmed according to the device operation log, the parameters of the audio channel corresponding to the new device are automatically adjusted, the audio signal in the switching process is ensured to be continuous and free of mutation, and the detailed switching operation is recorded in the log.

8. The audio data management method based on a sound console according to claim 1, characterized by, In the cloud tuning environment, the operation of the sound engineer is recorded as an independent operation step and version, so that engineers can collaboratively process the same audio, create and manage respective mix versions, support version comparison, merging, evaluation and visual analysis of parameter differences, including: The cloud server records and stores the operation steps and timestamps of the sound engineer, automatically generates mix versions, allows multiple engineers to collaboratively process and create independent version branches, supports version comparison to display the audio waveform, spectrum graph differences and corresponding operation differences of different mix versions, supports version merging to allow engineers to selectively merge operations and parameters from different version branches, automatically handle conflicts and provide conflict resolution suggestions, supports perceptual evaluation of generated mix versions, records evaluation results and feedback, and provides graphical display of parameter differences in the form of charts or timelines to visually analyze the differences between versions for engineers.

9. A sound console-based audio data management system using the sound console-based audio data management method according to any one of claims 1 to 8, characterized by, It comprises: A data acquisition and preprocessing module comprising a data acquisition unit and a data preprocessing unit; wherein the data acquisition unit acquires audio data, operation sequence data, device state and physical parameter data and spectral energy distribution data; the data preprocessing unit synchronizes the collected data in time, unifies the data format, and normalizes or standardizes numerical data; A feature extraction and correlation model establishment module comprising a feature extraction unit, a correlation model establishment unit and a model training unit; wherein the feature extraction unit extracts multi-dimensional features from the preprocessed audio data and operation sequence data to form feature vector sequences and time series feature vector sequences; the correlation model establishment unit inputs the feature sequences of audio and operation into LSTM to learn the time dynamic mode, and then inputs the LSTM output into Transformer to capture the complex interaction and long-distance dependence between features; a multi-modal fusion layer is designed to generate joint feature representation and build a correlation model; the model training unit trains the correlation model, receives real-time collected and preprocessed data, dynamically updates the state, and outputs real-time analysis results, audio data, operation sequence data and interrelated patterns; An abnormality detection and judgment module comprising an abnormality detection unit and a situation judgment unit; wherein the abnormality detection unit uses the trained correlation model to monitor audio data, operation sequence data, device state and physical parameter data and spectral energy distribution data in real time, and identifies abnormal data; the situation judgment unit judges the abnormal data found by the abnormality detection unit, and divides them into instrument failure, normal departure, microphone abnormality, audience seat noise and device switching. The abnormality processing module comprises an abnormality processing unit and a log recording unit; the abnormality processing unit executes corresponding processing strategies according to the judgment result of the condition judgment unit; and the log recording unit records the detailed information; The cloud collaboration and version management module comprises a version recording and branch management unit, a version comparison and merging unit and an evaluation feedback and visualization unit; the version recording and branch management unit records the operation steps and time stamps of the sound mixer on the cloud server, generates a mix version, supports multi-engineer collaboration, creates an independent version branch and works in parallel; the version comparison and merging unit displays the differences between different mix versions, supports version merging, allows engineers to selectively merge the operations of other branches, automatically handles conflicts and provides conflict resolution suggestions; and the evaluation feedback and visualization unit supports evaluation of the generated mix version, records the evaluation results and feedback and provides graphical display of parameter differences.

Images