Audio analysis method and system for difference positioning

Abstract

The invention relates to an audio analysis method and system for difference positioning, and relates to the field of audio processing, and the method comprises the steps: carrying out the comparison of audio physical characteristics, and obtaining a first undifferentiated audio moment; performing semantic deviation analysis on the first reference audio signal label and the first monitoring audio signal label to obtain a second undifferentiated audio moment; performing interval deviation analysis on the second reference audio signal label and the second monitoring audio signal label to obtain an interval difference audio moment; and adding the interval difference audio moment, the third reference audio signal label and the third monitoring audio signal label into an audio analysis result. Through the method and the device, the technical problem of insufficient accuracy and timeliness of audio difference positioning caused by the phenomena of heavy calculation burden, low analysis speed and easiness in noise interference in a traditional audio difference analysis method can be solved, unnecessary data processing burden can be effectively reduced through three-stage hierarchical analysis, and the positioning accuracy and timeliness of the audio difference are improved. And the audio difference positioning efficiency and precision are obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of audio processing, and in particular to an audio analysis method and system for difference positioning. Background Art

[0002] Audio difference analysis is to detect and locate the differences between two audio signals by comparing their features to identify deviations or anomalies. For example, in music teaching, real-time audio difference analysis can help students identify pitch, rhythm and harmony deviations in performance, and provide real-time feedback on wrong notes, rhythm deviations or inaccurate intervals, helping students correct problems in performance and improve practice results.

[0003] In the field of audio difference analysis, traditional methods usually rely on comparing and analyzing the global high-dimensional features of audio signals, involving multi-dimensional features such as frequency, timing, loudness, and pitch. Although these methods can identify audio differences to a certain extent, they also bring problems of heavy computational burden and slow analysis speed. Especially in high-noise environments, the robustness of the algorithm is significantly reduced. This method not only requires high-performance computing resource support, but also often fails to meet the accuracy and timeliness of audio difference positioning due to factors such as high data processing complexity and noise interference. Real-time application requirements cannot be met. Summary of the invention

[0004] The present invention aims to solve the technical problems that traditional audio difference analysis methods have heavy computational burden, slow analysis speed and susceptibility to noise interference, resulting in insufficient accuracy and timeliness of audio difference positioning. An audio analysis method and system for difference positioning are provided to solve the problem.

[0005] The technical solution of the present invention to solve the above technical problems is as follows: In a first aspect, the present invention provides an audio analysis method for difference positioning, which is applied to an audio analysis system for difference positioning, wherein the system includes a user end, the system is communicatively connected to an audio sensor, and the audio sensor is detachably installed on a sound source device, including: extracting audio physical features in response to a first audio signal uploaded by the user end to obtain reference audio feature timing information; extracting audio physical features of a second audio signal collected by the audio sensor to obtain monitoring audio feature timing information; comparing audio physical features of the reference audio feature timing information with the monitoring audio feature timing information to obtain a first indifference audio moment, wherein the first indifference audio moment has a first reference An audio signal tag and a first monitoring audio signal tag; performing semantic deviation analysis on the first reference audio signal tag and the first monitoring audio signal tag to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal tag and a second monitoring audio signal tag; performing interval deviation analysis on the second reference audio signal tag and the second monitoring audio signal tag to obtain an interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal tag and a third monitoring audio signal tag; adding the interval difference audio moment, the third reference audio signal tag and the third monitoring audio signal tag into the audio analysis result and sending it to the user end.

[0006] In a second aspect, the present invention provides an audio analysis system for difference positioning, the system comprising a user end, the system being communicatively connected to an audio sensor, the audio sensor being detachably mounted on a sound source device, comprising: a reference audio feature timing information acquisition module, for performing audio physical feature extraction in response to a first audio signal uploaded by the user end, and obtaining reference audio feature timing information; a monitoring audio feature timing information acquisition module, for receiving a second audio signal collected by the audio sensor, performing audio physical feature extraction, and obtaining monitoring audio feature timing information; an audio physical feature comparison module, for performing audio physical feature comparison on the reference audio feature timing information and the monitoring audio feature timing information, and obtaining a first indifference audio moment, wherein the first indifference audio moment has a first reference audio signal. A semantic deviation analysis module is used to perform semantic deviation analysis on the first reference audio signal label and the first monitoring audio signal label to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal label and a second monitoring audio signal label; an interval deviation analysis module is used to perform interval deviation analysis on the second reference audio signal label and the second monitoring audio signal label to obtain an interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal label and a third monitoring audio signal label; an audio analysis result sending module is used to add the interval difference audio moment, the third reference audio signal label and the third monitoring audio signal label into the audio analysis result and send it to the user end.

[0007] In a third aspect, the present invention further provides an electronic device, comprising: At least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can perform the steps of any one of the methods described in the first aspect above.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed, the steps of the method described in any one of the first aspects are implemented.

[0009] The beneficial effect of the present invention is that compared with the traditional analysis method of performing high-dimensional feature comparison on the entire audio segment, the present invention adopts a three-level hierarchical audio difference analysis method, which can perform physical feature comparison, semantic deviation analysis and interval deviation analysis on the audio signal layer by layer, and can gradually screen and accurately locate audio differences, thereby reducing unnecessary data processing burden, significantly improving the analysis accuracy and noise resistance of audio difference positioning, and can respond to audio difference detection needs more quickly, achieve accurate and fast audio difference positioning and feedback, and better adapt to application scenarios that require real-time analysis and high-precision feedback, such as music teaching, audio quality monitoring, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of a flow chart of an audio analysis method for difference localization provided by the present invention; Figure 2 A schematic diagram of the structure of an audio analysis system for difference positioning provided by the present invention; Figure 3 A schematic diagram of the structure of an electronic device provided by the present invention; Figure 4 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention.

[0011] In the accompanying drawings, the components represented by the reference numerals are described as follows: A reference audio feature timing information acquisition module 11, a monitoring audio feature timing information acquisition module 12, an audio physical feature comparison module 13, a semantic deviation analysis module 14, an interval deviation analysis module 15, an audio analysis result sending module 16, an electronic device 500, a memory 510, a processor 520, a first computer program 511, a computer-readable storage medium 600, and a second computer program 611. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0013] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0014] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0015] Embodiment 1, as Figure 1 As shown, an embodiment of the present invention provides an audio analysis method for difference positioning, which is applied to an audio analysis system for difference positioning, wherein the system includes a user end, the system is communicatively connected to an audio sensor, and the audio sensor is detachably installed on a sound source device, including: S100: extracting audio physical features in response to a first audio signal uploaded by a user terminal to obtain reference audio feature timing information.

[0016] Specifically, the method provided by the present invention is implemented through an audio analysis system for difference positioning, wherein the system includes a user end, the user end is used to receive and upload a reference audio signal to establish reference audio feature data, and receive audio difference analysis results fed back by the system, including specific feedback information such as physical feature deviation, semantic deviation and interval deviation, so as to help users quickly understand and locate deviation details in the audio; the system is communicatively connected to the audio sensor, that is, the system can transmit data and communicate in real time with the audio sensor in a wireless or wired manner, so as to obtain real-time data of the audio signal from the audio sensor and analyze and process it. This connection method allows the system to monitor changes in audio features in real time and promptly respond when an abnormality is detected. Feedback, wherein the audio sensor is designed to be flexibly loaded and unloaded to adapt to different sound source devices. Users can install the audio sensor on the specific device that needs to be monitored and change the device or position as needed. For example, the audio sensor can be installed on various devices such as musical instruments, broadcasting equipment or industrial machines by clamping, magnetic suction or bracket fixing to meet diverse application needs. For example, in music teaching, the audio sensor can be temporarily installed on musical instruments such as piano, guitar or violin to monitor the students' performance audio and compare it with the teacher's baseline audio through the system. After the real-time audio data collected by the audio sensor is transmitted to the system, the system analyzes the performance notes, rhythm and other characteristics, and feeds back the deviation to the students to help them quickly adjust their performance.

[0017] First, the audio analysis system receives a first audio signal uploaded by a user from a user terminal, that is, a reference audio file, such as an audio file of a standard performance. Then, the audio physical feature extraction is performed on the first audio signal, that is, the physical features of the first audio signal are extracted by an audio processing algorithm to obtain the core physical features of the first audio signal. The audio feature extraction can be performed by an existing algorithm, for example, using a Fourier transform algorithm to convert the audio signal from the time domain to the frequency domain to extract the frequency component; using a short-time energy analysis algorithm to calculate the energy change of the audio signal in different time windows to extract the volume feature; extracting the rhythm and duration features of the audio signal by autocorrelation analysis or zero crossing rate detection; obtaining the first physical feature extraction result, wherein the first physical feature extraction result includes frequency, loudness, duration, rhythm, and pitch, wherein the tone and harmony of the reference audio can be determined by frequency analysis, the abnormal volume fluctuation in comparison can be monitored by the loudness feature, the rhythm deviation can be identified by the rhythm and duration, and the accuracy of the note can be compared by the pitch. Then the first physical feature extraction result is arranged in time series to form the reference audio feature timing information. For example, in music teaching, the teacher uploads a standard piano performance audio, and generates the benchmark audio feature timing information of the performance by extracting the frequency, rhythm, pitch and loudness characteristics of the performance. Next, the system will compare the student's performance (monitoring audio) with this benchmark audio layer by layer to detect the student's performance deviation.

[0018] S200: Receive a second audio signal collected by an audio sensor to extract audio physical features and obtain monitoring audio feature timing information.

[0019] Specifically, the audio data is collected in real time by an audio sensor installed on the sound source device, and the collected second audio signal (monitoring audio) is sent to the audio analysis system through a wired or wireless communication connection. Then, the physical features of the second audio signal, including frequency, loudness, duration, rhythm, and pitch, are extracted through an audio processing algorithm to obtain a second physical feature extraction result, wherein the second physical feature extraction method is the same as the first physical feature extraction method described above. Then the second physical feature extraction results are arranged in chronological order to generate monitoring audio feature timing information. For example, in a music teaching scenario, an audio sensor collects students' real-time performance audio and uploads it to the audio analysis system. By extracting the physical features of the performance, such as frequency, loudness, and pitch, the feature timing information of the monitoring audio is generated. Next, the system compares this information with the baseline audio features of the standard performance, identifies deviations in notes or rhythms, and feeds the results back to the students so that they can correct errors in the performance in a timely manner.

[0020] S300: Performing an audio physical feature comparison on the reference audio feature timing information and the monitoring audio feature timing information to obtain a first indifference audio moment, wherein the first indifference audio moment has a first reference audio signal label and a first monitoring audio signal label.

[0021] In one embodiment, step S300 of the present application further includes: S310: Perform a global comparison of the audio physical features of the reference audio feature time series information and the monitoring audio feature time series information according to an audio physical feature comparison function to obtain a global deviation coefficient of the audio physical features.

[0022] In one embodiment, step S310 of the present application further includes: S311: Construct audio physical feature comparison function: , , , , , ,in, Characterize the reference audio feature timing information, ~ Characterize the time sequence information of Q attribute reference audio features, Characterize and monitor the audio feature timing information, ~ Characterize Q attributes to monitor the audio feature timing information, ~ Characterizes the time sequence information of the i-th attribute reference audio feature, ~ represents the i-th attribute monitoring audio feature timing information, N represents the timing number of the i-th attribute reference audio feature timing information, M represents the timing number of the i-th attribute monitoring audio feature timing information, Characterize the adjustment parameters, >1, Characterizes the physical characteristic deviation of the i-th attribute, Characterizes the global deviation coefficient of audio physical characteristics.

[0023] Specifically, first, an audio physical feature comparison function is constructed, in which: Characterizing the timing information of the reference audio features; ~ Characterize the time series information of Q attribute reference audio features, that is, the time series information of multiple attribute features (such as frequency, loudness, pitch, etc.) contained in the reference audio; Characterize and monitor audio feature timing information; ~ Characterize the time series information of Q attribute monitoring audio features, that is, the time series information of multiple attribute features (such as frequency, loudness, pitch, etc.) contained in the monitoring audio; ~ Characterizing the i-th attribute reference audio feature time sequence information, wherein the i-th attribute is any one of the Q attributes; ~ represents the characteristic timing information of the monitoring audio of the i-th attribute, N represents the timing number of the characteristic timing information of the reference audio of the i-th attribute, that is, the number of timing data points recorded by the reference audio on the i-th attribute; M represents the timing number of the characteristic timing information of the monitoring audio of the i-th attribute, that is, the number of timing data points recorded by the monitoring audio on the i-th attribute; Characterize the adjustment parameters, >1, among which, The default value is 10, which is used to adjust the degree of influence of timing deviation. For example, in a music teaching scenario, if the system is used to identify the rhythm deviation in the student's performance, adjusting the parameter It can be appropriately increased to ensure that even small timing deviations can be amplified and identified. In the device noise monitoring scenario, The value can be lowered to reduce the impact of timing deviations and focus more on frequency or loudness deviations; by adjusting the parameters Flexible settings can appropriately adjust the impact of timing deviation in different application scenarios to meet different analysis accuracy requirements; Characterizes the physical characteristic deviation of the i-th attribute, Characterizes the global deviation coefficient of audio physical characteristics and is used to comprehensively evaluate the reference audio and monitor the overall deviation of audio in all physical characteristics.

[0024] Then, according to the audio physical feature comparison function, a global audio physical feature comparison is performed on the reference audio feature time series information and the monitoring audio feature time series information, and the audio physical feature global deviation coefficient is calculated.

[0025] S320: When the global deviation coefficient of the audio physical feature is greater than or equal to the global deviation coefficient threshold, extract the first indifference audio moment and the physical feature difference audio moment, wherein the first indifference audio moment refers to the moment when each attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information are the same, and the physical feature difference audio moment refers to the moment when any attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information are different; S330: When the global deviation coefficient of the audio physical feature is less than the global deviation coefficient threshold, set all moments of the baseline audio feature timing information and the monitoring audio feature timing information as the first indifference audio moment.

[0026] Specifically, first, configure the global deviation coefficient threshold to determine whether there is a significant deviation in the overall physical characteristics of the reference audio and the monitoring audio. A reasonable global deviation coefficient threshold can be configured according to the application scenario and accuracy requirements. For example, in high-precision music teaching, the global deviation coefficient threshold can be set lower to identify subtle note and rhythm deviations. Among them, since a small amount of accidental errors may occur during the physical feature comparison process, if the global error is lower than the preset threshold, it is determined that the physical characteristics of the reference audio and the monitoring audio are consistent as a whole, and no further difference comparison is required; if the global error exceeds the threshold, it indicates that there is a significant difference between the reference audio and the monitoring audio, and it is necessary to conduct a detailed analysis and discretely extract the specific difference moments.

[0027] Then, the global deviation coefficient of the audio physical feature is judged according to the global deviation coefficient threshold. When the global deviation coefficient of the audio physical feature is greater than or equal to the global deviation coefficient threshold, the moment when each attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information are the same is extracted and set as the first indifference audio moment, that is, the time point when the baseline audio feature timing information and the monitoring audio feature timing information are the same in the audio physical feature of each attribute; and the moment when any attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information is different is extracted and set as the physical feature difference audio moment, that is, as long as there is a deviation in one physical feature (such as frequency or loudness), the moment will be marked as the physical feature difference audio moment, and the first indifference audio moment and the physical feature difference audio moment are obtained.

[0028] When the global deviation coefficient of the audio physical feature is less than the global deviation coefficient threshold, the physical features characterizing the reference audio and the monitoring audio are consistent as a whole, and all moments of the reference audio feature timing information and the monitoring audio feature timing information are set as the first indifference audio moments. By calculating the global deviation coefficient of the audio physical feature and setting the global deviation coefficient threshold for judgment, accidental errors can be quickly eliminated when judging the global deviation, and refined comparisons are only performed at the moment when the actual deviation is detected, thereby quickly screening audio clips with no significant differences, and concentrating computing resources on processing key difference moments, reducing the computing burden, and significantly improving the efficiency and accuracy of audio difference analysis, thereby providing users with more accurate and fast feedback.

[0029] S400: Performing semantic deviation analysis on the first reference audio signal label and the first monitoring audio signal label to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal label and a second monitoring audio signal label.

[0030] In one embodiment, step S400 of the present application further includes: S410: Extracting semantic features from the first reference audio signal label and the first monitoring audio signal label according to a semantic feature extraction model to obtain a first reference audio signal semantic feature and a first monitoring audio signal semantic feature.

[0031] In one embodiment, step S410 of the present application further includes: S411: configuring a semantic vector mapping space, wherein the semantic vector mapping space is used to map characters into unique multi-dimensional mathematical vectors; S412: constructing a semantic feature extraction loss function according to the semantic vector mapping space: ,in, Represents the semantic feature extraction loss value, Characterizes the adjustment coefficient, 0<b<1, represents the k-th dimension identity vector, Represent the k-th dimension extraction vector, H represents the dimension of the mathematical vector; S413: Taking the model of the sound source device as a constraint, collect the audio signal recording data set and the audio semantic identification data set of the same model of device; S414: Taking the audio semantic identification data set as supervision and the audio signal recording data set as input, train the semantic feature extraction model based on the semantic vector mapping space and the semantic feature extraction loss function.

[0032] Specifically, to construct a semantic feature extraction model, first, configure a semantic vector mapping space, wherein the semantic vector mapping space is used to map characters to unique multidimensional mathematical vectors, that is, configure a high-dimensional vector space (usually between 50 and 300 dimensions, and the specific dimension can be set according to the application scenario requirements) to carry the vector representation of each semantic unit, and map characters, words or notes to unique positions in the vector space through specific mapping methods (such as word embedding or character embedding). For example, using the audio semantic embedding method, each semantic unit is mapped to a corresponding multidimensional vector. For example, the character "do" can be mapped to a vector [0.12, 0.34, -0.45, ...], and "re" can be mapped to another unique vector. By configuring the semantic vector mapping space, abstract semantic features can be converted into operable vector representations, which not only facilitates rapid matching in subsequent semantic comparisons, but also ensures the uniqueness and discrimination of semantic features, thereby significantly improving the efficiency and accuracy of the system in difference detection, and can more accurately extract and compare semantic features in audio.

[0033] Then, a semantic feature extraction loss function is constructed according to the semantic vector mapping space. The loss function is used to optimize the semantic feature extraction model so that it can more accurately extract and represent semantic features, ensuring that the model generates accurate and highly discriminative vector representations in the mapping space. In the semantic feature extraction loss function, Represents the semantic feature extraction loss value, which is used to measure the accuracy of the model in extracting semantic features in the mapping space. The greater the similarity, the smaller the corresponding loss value; Characterize the adjustment coefficient. The range of the adjustment coefficient is 0<b<1. The adjustment coefficient is used to balance the weight of the loss function on the deviation of each dimensional vector, so that the model can flexibly adjust the feature extraction strength in different scenarios; represents the k-th dimension identity vector, represents the k-th dimension extraction vector, and H represents the dimension of the mathematical vector, that is, the total number of dimensions of the vector. This loss function quantifies the differences between similar semantic vectors to ensure that the greater the similarity, the smaller the loss, thereby guiding the model to generate more accurate semantic feature extraction vectors. At the same time, by introducing the adjustment coefficient to control the flexibility of the loss, it can further optimize the model's extraction accuracy and stability of semantic features, and is suitable for high-precision semantic comparison and deviation detection scenarios.

[0034] Further obtain the model of the sound source device (such as model number, series category, etc.), and use the model of the sound source device as the retrieval constraint feature to collect the audio signal recording data set and audio semantic identification data set of the same model device. Through this retrieval process, the device data irrelevant to the target model can be excluded to ensure that the data set only contains the audio signal data of the target model. Among them, the audio signal recording data and the audio semantic identification data correspond one-to-one, that is, each audio signal record has a corresponding audio semantic identification, which is used to accurately describe the characteristics and semantic content of the audio. The audio signal recording data includes the frequency, pitch, loudness, rhythm, interval and other physical feature information of the audio. The audio semantic identification data refers to the semantic description or label of the audio signal recording data, indicating the specific content or meaning contained in the audio signal. These identifications can be used to describe the state, notes, emotions, playing techniques, etc. of the audio signal, such as "C note", "G note", "glissando", "tremolo", etc.

[0035] A semantic feature extraction model is constructed based on a convolutional neural network. The semantic feature extraction model includes an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer, wherein the input data of the input layer is an audio signal, and the output data of the output layer is the semantic feature of the audio signal; low-level and high-level features in the spectrum are extracted by using multiple convolution layers, and each convolution layer extracts local features of the audio, such as frequency pattern, pitch change, etc., through the convolution kernel; the pooling layer reduces the data size by downsampling, reduces the computational complexity, and retains the main features; the fully connected layer can combine the local features extracted previously to generate a feature vector with more global significance.

[0036] Then, the audio signal recording data is used as input, the audio semantic identification data is used as supervision, based on the semantic vector mapping space and the semantic feature extraction loss function, the audio signal recording data set and the audio semantic identification data set are used as training data to perform supervised training on the semantic feature extraction model. First, the audio signal recording data set is converted into a format suitable for convolutional neural network processing (such as a spectrogram or a Mel-spectrogram) as the model input, and the audio semantic identification data set is mapped to the semantic vector space to generate a high-dimensional semantic vector as a supervision signal so that the model can learn the mapping relationship between audio features and semantic identification; then the preprocessed audio signal is input into the semantic feature extraction model, and the model is trained through a series of convolutional layers, pooling layers and The fully connected layer gradually extracts the physical and semantic features of the audio; then the distance between the model output semantic vector and the target semantic vector is calculated through the semantic feature extraction loss function, where the loss value reflects the similarity between the model's prediction result and the true semantic label, which is used to optimize the model; then the back propagation and gradient descent algorithms are used to update the model parameters (including convolution kernels and fully connected layer weights) according to the loss value. During the training process, the model continuously adjusts the parameters to gradually reduce the difference between the output semantic feature vector and the target semantic vector; then it is iterated multiple times on the entire dataset to gradually reduce the loss value, so that the model can obtain higher accuracy in semantic feature extraction until the expected convergence accuracy is met, and a trained semantic feature extraction model is obtained.

[0037] Finally, the first reference audio signal label and the first monitoring audio signal label are input into the semantic feature extraction model for semantic feature extraction, and the semantic features of the first reference audio signal and the semantic features of the first monitoring audio signal are output. By constructing a semantic feature extraction model, it is possible to effectively extract semantic information from complex audio signals, and at the same time improve the efficiency and accuracy of semantic information extraction, providing support for semantic deviation comparison.

[0038] S420: When the semantic feature of the first reference audio signal is the same as the semantic feature of the first monitoring audio signal, obtain the second indifference audio moment; S430: When the semantic feature of the first reference audio signal is different from the semantic feature of the first monitoring audio signal, obtain the semantic difference audio moment.

[0039] Specifically, when the semantic feature of the first reference audio signal is the same as the semantic feature of the first monitoring audio signal, and the semantic contents of the two audio segments at this moment are consistent, that is, there is no semantic difference between the reference audio and the monitoring audio during this period, then the moment is marked as a second indifference audio moment, added to the second indifference audio moment, and when all moments are analyzed, the second indifference audio moment is output, wherein the second indifference audio moment has a second reference audio signal label and a second monitoring audio signal label; when the semantic feature of the first reference audio signal is different from the semantic feature of the first monitoring audio signal, and there is a difference in the semantic contents of the two audio segments at this moment, then the moment is marked as a semantic difference audio moment, and the semantic difference audio moment is obtained.

[0040] S500: Performing interval deviation analysis on the second reference audio signal label and the second monitoring audio signal label to obtain an interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal label and a third monitoring audio signal label.

[0041] In one embodiment, step S500 of the present application further includes: S510: Extracting the reference pitch and reference frequency of the second reference audio signal label; S520: Extracting the monitoring pitch and monitoring frequency of the second monitoring audio signal label; S530: When the reference pitch and the monitoring pitch are different, or / and the reference frequency and the monitoring frequency are different, obtaining the interval difference audio moment.

[0042] Specifically, first, extract the reference pitch and reference frequency of the moment from the second reference audio signal tag as the reference interval information of the moment, the reference pitch represents the tone of the reference audio, and the reference frequency represents the frequency of the tone, and this information is used as a reference reference for interval comparison; then extract the monitoring pitch and monitoring frequency of the moment from the second monitoring audio signal tag; then compare the reference pitch with the monitoring pitch, and the reference frequency with the monitoring frequency, and when the reference pitch and the monitoring pitch are different and / or the reference frequency and the monitoring frequency are different, set the moment as the interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal tag and a third monitoring audio signal tag. By determining the interval difference based on pitch and frequency, the interval difference audio moment can be efficiently and accurately identified, providing support for audio difference positioning.

[0043] S600: Add the interval difference audio moment, the third reference audio signal tag and the third monitoring audio signal tag to the audio analysis result and send it to the user end.

[0044] In one embodiment, step S600 of the present application further includes: S610: Obtain the physical feature difference audio moment of the audio physical feature comparison between the baseline audio feature timing information and the monitoring audio feature timing information, wherein the physical feature difference audio moment has a fourth baseline audio signal label and a fourth monitoring audio signal label; S620: Obtain the semantic difference audio moment of the semantic deviation analysis of the first baseline audio signal label and the first monitoring audio signal label, wherein the semantic difference audio moment has a fifth baseline audio signal label and a fifth monitoring audio signal label; S630: Construct an interval deviation ternary array according to the interval difference audio moment, the third baseline audio signal label and the third monitoring audio signal label, construct a physical feature deviation ternary array according to the physical feature difference audio moment, the fourth baseline audio signal label and the fourth monitoring audio signal label, and construct a semantic deviation ternary array according to the semantic difference audio moment, the fifth baseline audio signal label and the fifth monitoring audio signal label; S640: Add the interval deviation ternary array, the physical feature deviation ternary array and the semantic deviation ternary array into the audio analysis result and send them to the user end.

[0045] Specifically, the moment when there is a difference in the audio physical feature comparison process between the reference audio feature timing information and the monitoring audio feature timing information is set as the physical feature difference audio moment, and the physical feature difference audio moment is obtained, wherein the physical feature difference audio moment has a fourth reference audio signal label and a fourth monitoring audio signal label. The moment when there is a difference in the semantic deviation analysis process between the first reference audio signal label and the first monitoring audio signal label is set as the semantic difference audio moment, wherein the semantic difference audio moment has a fifth reference audio signal label and a fifth monitoring audio signal label.

[0046] Then, according to the interval difference audio moment, the third reference audio signal label and the third monitoring audio signal label, a three-element array of interval deviation is constructed, that is, for each interval difference audio moment, it is combined with the corresponding third reference audio signal label and the third monitoring audio signal label to form a three-element array of interval deviation. According to the physical feature difference audio moment, the fourth reference audio signal label and the fourth monitoring audio signal label, a three-element array of physical feature deviation is constructed, that is, for each physical feature difference audio moment, it is combined with the corresponding fourth reference audio signal label and the fourth monitoring audio signal label to form a three-element array of physical feature deviation. According to the semantic difference audio moment, the fifth reference audio signal label and the fifth monitoring audio signal label, a three-element array of semantic deviation is constructed, that is, for each semantic difference audio moment, it is combined with the corresponding fifth reference audio signal label and the fifth monitoring audio signal label to form a three-element array of semantic deviation.

[0047] Finally, the pitch deviation ternary array, the physical feature deviation ternary array and the semantic deviation ternary array are integrated into a complete audio analysis result, and the audio analysis result is sent to the user end. By sending the pitch, physical feature and semantic deviation ternary arrays as audio analysis results to the user end, detailed multi-dimensional difference analysis information can be provided to the user, so that the user can fully understand the differences in audio signals and make efficient decisions and adjustments.

[0048] An audio analysis method for difference localization provided by an embodiment of the present invention has at least the following technical effects: 1. By adopting a three-level hierarchical audio difference analysis method, the physical feature comparison, semantic deviation analysis and interval deviation analysis of the audio signal can be performed layer by layer, and the audio differences can be gradually screened and accurately located, thereby reducing unnecessary data processing burden, significantly improving the analysis accuracy and noise resistance of audio difference positioning, and being able to respond to audio difference detection needs more quickly, achieving accurate and fast audio difference positioning and feedback, and better adapting to application scenarios that require real-time analysis and high-precision feedback, such as music teaching, audio quality monitoring, etc.

[0049] 2. By calculating the global deviation coefficient of the audio physical characteristics and setting the global deviation coefficient threshold for judgment, accidental errors can be quickly eliminated when judging the global deviation, and refined comparison is only performed when the actual deviation is detected, so as to quickly screen audio clips with no significant differences and concentrate computing resources on processing key difference moments, reducing the computing burden and significantly improving the efficiency and accuracy of audio difference analysis, thereby providing users with more accurate and fast feedback.

[0050] Embodiment 2, as Figure 2 As shown, based on the same inventive concept as the audio analysis method for difference positioning provided in the first embodiment, the embodiment of the present invention further provides an audio analysis system for difference positioning, the system comprising a user end, the system and an audio sensor are communicatively connected, the audio sensor is detachably mounted on a sound source device, and comprises: The reference audio feature timing information acquisition module 11 is used to extract audio physical features in response to the first audio signal uploaded by the user end, and obtain the reference audio feature timing information; the monitoring audio feature timing information acquisition module 12 is used to receive the second audio signal collected by the audio sensor to extract audio physical features and obtain the monitoring audio feature timing information; the audio physical feature comparison module 13 is used to compare the reference audio feature timing information with the monitoring audio feature timing information to obtain the first indifference audio moment, wherein the first indifference audio moment has a first reference audio signal label and a first monitoring audio signal label; the semantic deviation analysis module 14 is used to compare the first reference audio A signal tag and the first monitoring audio signal tag are subjected to semantic deviation analysis to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal tag and a second monitoring audio signal tag; an interval deviation analysis module 15 is used to perform interval deviation analysis on the second reference audio signal tag and the second monitoring audio signal tag to obtain an interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal tag and a third monitoring audio signal tag; an audio analysis result sending module 16 is used to add the interval difference audio moment, the third reference audio signal tag and the third monitoring audio signal tag into the audio analysis result and send it to the user end.

[0051] In one embodiment, the audio analysis system for difference location further includes: Obtain a physical feature difference audio moment of audio physical feature comparison between the baseline audio feature timing information and the monitoring audio feature timing information, wherein the physical feature difference audio moment has a fourth baseline audio signal label and a fourth monitoring audio signal label; obtain a semantic difference audio moment of semantic deviation analysis of the first baseline audio signal label and the first monitoring audio signal label, wherein the semantic difference audio moment has a fifth baseline audio signal label and a fifth monitoring audio signal label; construct an interval deviation ternary array according to the interval difference audio moment, the third baseline audio signal label and the third monitoring audio signal label, construct a physical feature deviation ternary array according to the physical feature difference audio moment, the fourth baseline audio signal label and the fourth monitoring audio signal label, and construct a semantic deviation ternary array according to the semantic difference audio moment, the fifth baseline audio signal label and the fifth monitoring audio signal label; add the interval deviation ternary array, the physical feature deviation ternary array and the semantic deviation ternary array into the audio analysis result and send them to the user end.

[0052] In one embodiment, the audio analysis system for difference location further includes: A global comparison of audio physical features of the baseline audio feature timing information and the monitoring audio feature timing information is performed according to the audio physical feature comparison function to obtain a global deviation coefficient of the audio physical features; when the global deviation coefficient of the audio physical features is greater than or equal to a global deviation coefficient threshold, the first indifference audio moment and the audio moment with physical feature difference are extracted, wherein the first indifference audio moment refers to a moment when each attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information is the same, and the audio moment with physical feature difference refers to a moment when any attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information is different; when the global deviation coefficient of the audio physical features is less than the global deviation coefficient threshold, all moments of the baseline audio feature timing information and the monitoring audio feature timing information are set as the first indifference audio moment.

[0053] In one embodiment, the audio analysis system for difference location further includes: According to the semantic feature extraction model, semantic features are extracted from the first reference audio signal label and the first monitoring audio signal label to obtain the first reference audio signal semantic feature and the first monitoring audio signal semantic feature; when the first reference audio signal semantic feature and the first monitoring audio signal semantic feature are the same, the second indifference audio moment is obtained; when the first reference audio signal semantic feature and the first monitoring audio signal semantic feature are different, the semantic difference audio moment is obtained.

[0054] In one embodiment, the audio analysis system for difference location further includes: A semantic vector mapping space is configured, where the semantic vector mapping space is used to map characters into unique multi-dimensional mathematical vectors; and a semantic feature extraction loss function is constructed according to the semantic vector mapping space: ,in, Represents the semantic feature extraction loss value, Characterizes the adjustment coefficient, 0<b<1, represents the k-th dimension identity vector, The k-th dimension extraction vector is represented, and H represents the dimension of the mathematical vector; taking the model of the sound source device as a constraint, an audio signal recording data set and an audio semantic identification data set of the same model of the device are collected; taking the audio semantic identification data set as supervision and the audio signal recording data set as input, the semantic feature extraction model is trained based on the semantic vector mapping space and the semantic feature extraction loss function.

[0055] In one embodiment, the audio analysis system for difference location further includes: Extracting a reference pitch and a reference frequency of the second reference audio signal label; extracting a monitoring pitch and a monitoring frequency of the second monitoring audio signal label; when the reference pitch is different from the monitoring pitch, or / and the reference frequency is different from the monitoring frequency, obtaining the interval difference audio moment.

[0056] In one embodiment, the audio analysis system for difference location further includes: Construct an audio physical feature comparison function: , , , , , ,in, Characterize the reference audio feature timing information, ~ Characterize the time sequence information of Q attribute reference audio features, Characterize and monitor the audio feature timing information, ~ Characterize Q attributes to monitor the audio feature timing information, ~ Characterizes the time sequence information of the i-th attribute reference audio feature, ~ represents the i-th attribute monitoring audio feature timing information, N represents the timing number of the i-th attribute reference audio feature timing information, M represents the timing number of the i-th attribute monitoring audio feature timing information, Characterize the adjustment parameters, >1, Characterizes the physical characteristic deviation of the i-th attribute, Characterizes the global deviation coefficient of audio physical characteristics.

[0057] For example 3, please refer to Figure 3 , Figure 3 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 3As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a first computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the first computer program 511, the following steps are implemented: performing audio physical feature extraction in response to a first audio signal uploaded by a user end to obtain reference audio feature timing information; receiving a second audio signal collected by an audio sensor to perform audio physical feature extraction to obtain monitoring audio feature timing information; performing audio physical feature comparison on the reference audio feature timing information and the monitoring audio feature timing information to obtain a first indifference audio moment, wherein the first indifference audio moment has a first A reference audio signal tag and a first monitoring audio signal tag; performing semantic deviation analysis on the first reference audio signal tag and the first monitoring audio signal tag to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal tag and a second monitoring audio signal tag; performing interval deviation analysis on the second reference audio signal tag and the second monitoring audio signal tag to obtain an interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal tag and a third monitoring audio signal tag; adding the interval difference audio moment, the third reference audio signal tag and the third monitoring audio signal tag into the audio analysis result and sending it to the user end.

[0058] Example 4, please refer to Figure 4 , Figure 4 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present invention. Figure 4As shown, this embodiment provides a computer-readable storage medium 600, on which a second computer program 611 is stored. When the second computer program 611 is executed by a processor, the following steps are implemented: extracting audio physical features in response to a first audio signal uploaded by a user end to obtain baseline audio feature timing information; extracting audio physical features of a second audio signal collected by an audio sensor to obtain monitoring audio feature timing information; performing audio physical feature comparison on the baseline audio feature timing information and the monitoring audio feature timing information to obtain a first indifference audio moment, wherein the first indifference audio moment has a first baseline audio signal label and a first monitoring audio signal label. signal tag; performing semantic deviation analysis on the first reference audio signal tag and the first monitoring audio signal tag to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal tag and a second monitoring audio signal tag; performing interval deviation analysis on the second reference audio signal tag and the second monitoring audio signal tag to obtain an interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal tag and a third monitoring audio signal tag; adding the interval difference audio moment, the third reference audio signal tag and the third monitoring audio signal tag into the audio analysis result and sending it to the user end.

[0059] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0060] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0062] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0064] Although preferred embodiments of the present invention have been described, additional changes and modifications may occur to these embodiments once those skilled in the art understand the basic inventive concepts.

[0065] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention belong to the scope of the present invention and its equivalent technologies, the present invention is also intended to include these changes and variations.

Claims

1. An audio analysis method for difference localization, characterized in that: Applied to an audio analysis system for difference positioning, the system includes a user end, the system is communicatively connected to an audio sensor, and the audio sensor is detachably mounted on a sound source device, including: Extracting audio physical features in response to a first audio signal uploaded by a user terminal to obtain reference audio feature timing information; Receive the second audio signal collected by the audio sensor to extract audio physical features and obtain monitoring audio feature timing information; Performing audio physical feature comparison on the reference audio feature timing information and the monitoring audio feature timing information to obtain a first indifference audio moment, wherein the first indifference audio moment has a first reference audio signal label and a first monitoring audio signal label; Performing semantic deviation analysis on the first reference audio signal label and the first monitoring audio signal label to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal label and a second monitoring audio signal label; Performing interval deviation analysis on the second reference audio signal label and the second monitoring audio signal label to obtain an interval difference audio moment, wherein the interval difference audio moment has a third reference audio signal label and a third monitoring audio signal label; The interval difference audio moment, the third reference audio signal tag and the third monitoring audio signal tag are added to the audio analysis result and sent to the user end.

2. The method according to claim 1, characterized in that The step of adding the interval difference audio time, the third reference audio signal tag and the third monitoring audio signal tag into the audio analysis result and sending it to the user end further includes: Obtaining a physical feature difference audio moment of audio physical feature comparison between the reference audio feature timing information and the monitoring audio feature timing information, wherein the physical feature difference audio moment has a fourth reference audio signal label and a fourth monitoring audio signal label; Obtaining a semantic difference audio moment of a semantic deviation analysis of the first reference audio signal label and the first monitoring audio signal label, wherein the semantic difference audio moment has a fifth reference audio signal label and a fifth monitoring audio signal label; Constructing a three-element array of interval deviations according to the interval difference audio moment, the third reference audio signal label, and the third monitoring audio signal label; constructing a three-element array of physical feature deviations according to the physical feature difference audio moment, the fourth reference audio signal label, and the fourth monitoring audio signal label; and constructing a three-element array of semantic deviations according to the semantic difference audio moment, the fifth reference audio signal label, and the fifth monitoring audio signal label; The pitch deviation ternary array, the physical feature deviation ternary array and the semantic deviation ternary array are added to the audio analysis result and sent to the user end.

3. The method according to claim 1, characterized in that Performing audio physical feature comparison on the reference audio feature time sequence information and the monitoring audio feature time sequence information to obtain a first indifference audio moment includes: Performing a global comparison of audio physical features on the reference audio feature time series information and the monitoring audio feature time series information according to an audio physical feature comparison function to obtain a global deviation coefficient of the audio physical feature; When the global deviation coefficient of the audio physical feature is greater than or equal to the global deviation coefficient threshold, the first indifference audio moment and the physical feature difference audio moment are extracted, wherein the first indifference audio moment refers to the moment when each attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information is the same, and the physical feature difference audio moment refers to the moment when any attribute audio physical feature of the baseline audio feature timing information and the monitoring audio feature timing information is different; When the global deviation coefficient of the audio physical feature is less than the global deviation coefficient threshold, all moments of the reference audio feature timing information and the monitoring audio feature timing information are set as the first indifference audio moments.

4. The method according to claim 1, characterized in that Performing semantic deviation analysis on the first reference audio signal label and the first monitoring audio signal label to obtain a second indifferent audio moment includes: According to the semantic feature extraction model, semantic features are extracted from the first reference audio signal label and the first monitoring audio signal label to obtain a first reference audio signal semantic feature and a first monitoring audio signal semantic feature; When the semantic feature of the first reference audio signal is the same as the semantic feature of the first monitoring audio signal, obtaining the second indifference audio moment; When the semantic feature of the first reference audio signal is different from the semantic feature of the first monitoring audio signal, a semantic difference audio moment is obtained.

5. The method according to claim 4, characterized in that The semantic feature extraction model construction step includes: Configuring a semantic vector mapping space, wherein the semantic vector mapping space is used to map characters into unique multi-dimensional mathematical vectors; According to the semantic vector mapping space, a semantic feature extraction loss function is constructed: ， in, Represents the semantic feature extraction loss value, Characterizes the adjustment coefficient, 0<b<1, represents the k-th dimension identity vector, represents the k-th dimension extraction vector, and H represents the dimension of the mathematical vector; Taking the audio source device model as a constraint, collect the audio signal recording dataset and audio semantic identification dataset of the same model of device; The semantic feature extraction model is trained based on the semantic vector mapping space and the semantic feature extraction loss function, with the audio semantic identification dataset as supervision and the audio signal recording dataset as input.

6. The method according to claim 1, characterized in that Performing interval deviation analysis on the second reference audio signal label and the second monitoring audio signal label to obtain interval difference audio moments includes: Extracting a reference pitch and a reference frequency of the second reference audio signal label; Extracting the monitoring pitch and monitoring frequency of the second monitoring audio signal label; When the reference pitch is different from the monitoring pitch, or / and the reference frequency is different from the monitoring frequency, the interval difference audio moment is obtained.

7. The method according to claim 3, characterized in that The audio physical feature comparison function is: Construct an audio physical feature comparison function: ， ， ， ， ， ， in, Characterize the reference audio feature timing information, ~ Characterize the time sequence information of Q attribute reference audio features, Characterize and monitor the audio feature timing information, ~ Characterize Q attributes to monitor the audio feature timing information, ~ Characterizes the time sequence information of the i-th attribute reference audio feature, ~ represents the i-th attribute monitoring audio feature timing information, N represents the timing number of the i-th attribute reference audio feature timing information, M represents the timing number of the i-th attribute monitoring audio feature timing information, Characterize the adjustment parameters, >1, Characterizes the physical characteristic deviation of the i-th attribute, Characterizes the global deviation coefficient of audio physical characteristics.

8. An audio analysis system for difference localization, characterized in that: The steps for implementing the audio analysis method for difference positioning according to any one of claims 1 to 7, the system comprising a user end, the system being communicatively connected to an audio sensor, the audio sensor being detachably mounted on a sound source device, comprising: A reference audio feature timing information acquisition module, configured to extract audio physical features in response to a first audio signal uploaded by a user terminal, and obtain reference audio feature timing information; A monitoring audio feature timing information acquisition module is used to receive a second audio signal collected by an audio sensor to extract audio physical features and obtain monitoring audio feature timing information; An audio physical feature comparison module, used for performing audio physical feature comparison on the reference audio feature timing information and the monitoring audio feature timing information to obtain a first indifference audio moment, wherein the first indifference audio moment has a first reference audio signal label and a first monitoring audio signal label; A semantic deviation analysis module, configured to perform semantic deviation analysis on the first reference audio signal label and the first monitoring audio signal label to obtain a second indifference audio moment, wherein the second indifference audio moment has a second reference audio signal label and a second monitoring audio signal label; a pitch deviation analysis module, configured to perform pitch deviation analysis on the second reference audio signal label and the second monitoring audio signal label to obtain a pitch difference audio moment, wherein the pitch difference audio moment has a third reference audio signal label and a third monitoring audio signal label; The audio analysis result sending module is used to add the interval difference audio moment, the third reference audio signal label and the third monitoring audio signal label into the audio analysis result and send it to the user end.

9. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the audio analysis method for difference positioning as described in claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, the audio analysis method for difference positioning as described in claims 1 to 7 is implemented.

Images