A method and device for sound recognition and positioning
Through the joint classification regression model, the prediction of sound source categories and distances using FBank and LPC features is solved, and the problems of low sound source positioning accuracy and high calculation cost in the prior art are achieved, and the sound source positioning with high accuracy and low complexity are achieved.
Patent Information
- Application Number
- CN202411345733.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-26
AI Technical Summary
The existing sound source positioning technology has problems with low positioning accuracy and high calculation cost, especially the timing characteristics of the audio signal and the failure to select appropriate input characteristics, resulting in low positioning accuracy.
Using a joint classification regression model, multi-model splicing features are generated by extracting FBank features and LPC features of audio data, and modeling them to predict the sound source category and sound source distance, thereby determining the location of user equipment and reducing the computational complexity. degree and cost.
High-precision sound source positioning is achieved, reducing the computational complexity and cost, and it only needs to obtain three audio data to accurately locate the user equipment, improving the positioning effect.
Smart Images

Figure CN118884350B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of sound positioning, and in particular to a method and device for sound recognition and positioning. Background Art
[0002] Since the Global Positioning System (GPS) has many practical application problems in indoor environments and cannot be used, acoustic positioning technology is generally used to achieve indoor positioning. Among acoustic positioning technologies, sound source positioning has broad application prospects and practical significance. For example, smart home robots need to locate human voices during human-computer interaction. Based on the detection and positioning of human voices, they change their field of vision and then use computer vision to accurately locate the object they want to interact with, ultimately achieving intelligent interaction of voice and video.
[0003] The existing sound source localization solutions often have the problems of low positioning accuracy and high computational cost. Specifically: (1) Too many features of the original audio signal emitted by the sound source device need to be used, which leads to complex calculations and poor practicality; (2) The temporal features of the original audio signal are not utilized, resulting in low positioning accuracy; (3) In the positioning solution based on machine learning, the positioning accuracy is low due to reasons such as failure to select better input features.
[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a method and device for sound recognition and positioning, the purpose of which is to provide a joint classification regression model, and based on it, predict the sound source category and sound source distance of at least three sound source devices, thereby realizing the positioning of user equipment, with high positioning accuracy while ensuring positioning accuracy, solving the problems of low positioning accuracy and high computational cost in the sound source positioning solution of the prior art.
[0006] The present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides a method for sound recognition and positioning, comprising:
[0008] Extract FBank features and LPC features of at least three audio data;
[0009] According to the FBank feature and the LPC feature of each audio data, a multi-model splicing feature of the audio data is obtained; a pattern related to the sound source category and the sound source distance in the multi-model splicing feature is modeled to obtain a joint feature representation; wherein the sound source distance is the distance between the sound source device and the user device, and the user device is used to collect audio data of the at least three sound source devices;
[0010] Predicting the sound source category to which the audio data belongs based on the joint feature representation; predicting the sound source distance of the audio data based on the joint feature representation;
[0011] The position of the user equipment is determined according to the sound source categories and sound source distances of at least three audio data.
[0012] Further, the multi-model concatenation feature of the audio data is obtained according to the FBank feature and the LPC feature of each audio data; the pattern related to the sound source category and the sound source distance in the multi-model concatenation feature is modeled to obtain the joint feature representation, including:
[0013] Adaptively average pooling the FBank features to obtain compressed features;
[0014] According to a preset splicing method, the compressed features and the LPC features are stacked in the channel dimension to obtain the multi-model splicing features;
[0015] Inputting the multi-model concatenation feature into a plurality of cascaded audio sequence modeling modules, wherein the input feature of the first audio sequence modeling module is the multi-model concatenation feature; the input feature of the other audio sequence modeling modules is the output feature of the previous audio sequence modeling module, and the joint feature representation is obtained from the last audio sequence modeling module;
[0016] Among them, the execution logic of each audio sequence modeling module is: learn the dependency of sound source category and / or sound source distance from the input features, obtain output features that characterize the dependency, and input the output features to the next audio sequence modeling module so that the next audio sequence modeling module can update the currently learned dependency.
[0017] Furthermore, learning the dependency relationship of the sound source category and / or the sound source distance from the input features to obtain the output features representing the dependency relationship includes:
[0018] Performing a linear transformation on the input feature to expand the dimension of the input feature to obtain an increased-dimensional feature representation;
[0019] The nonlinear features of the dimension-increased feature representation are learned using a first activation function to obtain a first nonlinear feature representation; the expression of the first activation function is: ; Wherein, a is the feature input into the first activation function;
[0020] Performing a convolution operation on the increased-dimensional feature representation to extract local features of the increased-dimensional feature representation to obtain a local feature representation; using a first activation function to learn nonlinear features of the local feature representation to obtain a second nonlinear feature representation; assigning corresponding attention weights to each feature in the second nonlinear feature representation, compressing the second nonlinear feature representation based on the attention weights, and extracting a third nonlinear feature representation;
[0021] The first nonlinear feature representation and the third nonlinear feature representation are nonlinearly connected to obtain output features.
[0022] Further, predicting the sound source category to which the audio data belongs according to the joint feature representation includes:
[0023] Linearly mapping the joint feature representation into a two-dimensional feature representation;
[0024] The nonlinear features of the two-dimensional feature representation are learned using a second activation function to obtain a classification feature representation; the expression of the second activation function is: ; Wherein, b is the feature input into the second activation function;
[0025] Linearly mapping the classification feature representation into a class probability distribution;
[0026] According to a preset selection rule, the sound source category is determined from the category probability distribution.
[0027] Further, predicting the sound source distance of the audio data according to the joint feature representation includes:
[0028] Linearly mapping the joint feature representation into a one-dimensional feature representation;
[0029] The nonlinear features of the one-dimensional feature representation are learned using a third activation function to obtain an intermediate regression feature representation; the expression of the third activation function is: ; Wherein, c is the feature of the third activation function input;
[0030] Using a third activation function to learn the nonlinear features of the intermediate regression feature representation to obtain a target regression feature representation;
[0031] The target regression feature representation is linearly mapped to the sound source distance.
[0032] Further, determining the location of the user equipment according to the sound source category and the sound source distance of at least three audio data includes:
[0033] For each of the at least three sound source devices, taking the first position coordinate of the sound source device as the center of the corresponding three-dimensional audio circle, taking the sound source distance corresponding to the sound source device as the radius of the three-dimensional audio circle, and constructing the three-dimensional audio circle;
[0034] The difference between the user's horizontal coordinate to be solved and the sound source horizontal coordinate of each sound source device is used as the corresponding horizontal distance; the difference between the user's vertical coordinate to be solved and the sound source vertical coordinate of each sound source device is used as the corresponding vertical distance; and a spatial position relationship between the horizontal distance and the vertical distance corresponding to each three-dimensional audio circle and the corresponding sound source distance is constructed;
[0035] The spatial position relationship equation is solved to obtain the user horizontal coordinate and the user vertical coordinate to determine the position of the user equipment.
[0036] Further, solving the spatial position relationship equation to obtain the user horizontal coordinate and the user vertical coordinate includes:
[0037] Solving the spatial position relationship equation to obtain the user horizontal coordinate and the user vertical coordinate includes:
[0038] Each two equations in the spatial position relationship are regarded as a system of equations, and the solution of the system of equations is set as the initial position of the particle; where the solution of each system of equations is ( ), ; The initial velocity of particles is 0;
[0039] Based on the initial position and the initial velocity, the objective function value of each particle is iteratively calculated according to the objective function; wherein the expression of the objective function is: , is the sound source horizontal coordinate of the i-th sound source device, is the sound source ordinate of the i-th sound source device, is the sound source distance of the i-th sound source device;
[0040] If the objective function value of the particle's new position is greater than the objective function value of its individual optimal position, the individual optimal position is updated using the new position; if the objective function value of the new position is greater than the objective function value of the global optimal position, the global optimal position is updated using the new position; in each iteration, for the For each particle, update its velocity and position using the following formula:
[0041] +
[0042] +
[0043]
[0044]
[0045] in, For the The vertical velocity of a particle is For the The velocity of a particle in the horizontal direction, and the inertia weight is , is a learning factor, is another learning factor, is a random seed, is another random seed; The new position of the particle is ( , ), No. The individual optimal position of a particle is ( , ), No. The global optimal position of a particle is ( , );
[0046] When the maximum number of iterations is reached or the change in the objective function value of the global optimal position is less than the convergence criterion, the iteration is terminated; the horizontal coordinate of the global optimal position after the termination of the iteration is used as the user horizontal coordinate, and the vertical coordinate of the global optimal position after the termination of the iteration is used as the user vertical coordinate.
[0047] Furthermore, before extracting the FBank features and LPC features of at least three audio data, speech enhancement is performed on the original audio signal to obtain corresponding audio data;
[0048] The method for sound recognition and positioning also includes:
[0049] Obtaining a singular value corresponding to the original audio signal through an SVD algorithm; obtaining an estimate of the number of sound sources according to the singular value and the corresponding attenuation rate; and processing the original audio signal through a FastICA algorithm according to the estimate of the number of sound sources to obtain at least one independent sound source signal;
[0050] The independent sound source signal is encoded using the WavLM model in a self-supervised learning manner to extract the original continuous features therein; the original continuous features are clustered using a K-Means clustering algorithm to convert the original continuous features into initial discrete representations;
[0051] A Transformer-based Decoder-Only structure language model is used to extract context information from the initial discrete representation, model the initial discrete representation based on the context information, and predict a target discrete representation after speech enhancement;
[0052] The target discrete representation is converted back to the target continuous feature through a deserializer; the target continuous feature is restored to the sound source sample to be judged through a HiFi-GAN model;
[0053] Determine the cosine distance between the sound source sample to be determined and a preset sound source sample in a sound source library, and obtain the similarity between the sound source sample to be determined and the preset sound source sample based on the cosine distance; when the similarity is greater than a similarity threshold, determine the sound source sample to be determined as audio data.
[0054] In a second aspect, the present invention further provides a device for sound recognition and positioning, which is used to implement the method for sound recognition and positioning described in the first aspect, and the device for sound recognition and positioning includes:
[0055] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to perform the method for sound recognition and positioning described in the first aspect.
[0056] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method for sound recognition and positioning described in the first aspect.
[0057] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in a memory to execute the method for sound recognition and positioning in the first aspect.
[0058] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, enables the computer or the processor to execute the method for sound recognition and positioning as described in the first to fourth aspects and any one of the above.
[0059] In a sixth aspect, a method system for sound recognition and positioning is provided, comprising a device for sound recognition and positioning as in the second aspect, and using the method for sound recognition and positioning as described in the first aspect to complete the interaction with the device for sound recognition and positioning of the second aspect.
[0060] Different from the prior art, the present invention has at least the following beneficial effects:
[0061] The present invention extracts FBank features and LPC features of at least three audio data, generates multi-model splicing features of audio data based on them, and models the patterns related to the sound source category and sound source distance therein to obtain a joint feature representation, thereby respectively realizing the prediction of the sound source category and sound source distance to which the audio data belongs; finally, according to the sound source category and sound source distance of at least three audio data, at least three three-dimensional audio spheres are constructed, and the intersection of the constructed multiple three-dimensional audio spheres is determined as the position of the user device. Among them, the present invention only uses FBank features and LPC features, which greatly reduces the calculation complexity and cost, and has high practicality; by jointly analyzing the sound source category and sound source distance, the positioning effect is greatly improved in combination with the time series features; only at least three audio data need to be obtained to realize the positioning of the user device, and the positioning accuracy is high. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0063] Figure 1 It is a flow chart of a method for sound recognition and positioning provided by an embodiment of the present invention;
[0064] Figure 2 This is a schematic diagram of a use scenario of a method for voice recognition and positioning according to an embodiment of the present invention;
[0065] Figure 3 is a schematic diagram of a usage scenario of another method for voice recognition and positioning according to an embodiment of the present invention;
[0066] Figure 4 It is a flowchart of performing speech enhancement before step 10 provided by an embodiment of the present invention;
[0067] Figure 5 is a flow chart of step 40 provided in an embodiment of the present invention;
[0068] Figure 6 is another flow chart of step 40 provided in an embodiment of the present invention;
[0069] Figure 7 is a flow chart of step 20 provided in an embodiment of the present invention;
[0070] Figure 8Schematic diagram of a network structure of a joint classification and regression network provided by an embodiment of the present invention;
[0071] Fig. 9 is a flow chart of step 203 provided by an embodiment of the present invention;
[0072] Fig.10 is a schematic diagram of a network structure of an audio sequence modeling module provided by an embodiment of the present invention;
[0073] Fig.11 is a flow chart of step 30 provided in an embodiment of the present invention;
[0074] Fig.12 is another flowchart of step 30 provided in an embodiment of the present invention;
[0075] Fig.13 It is a schematic diagram of the architecture of a device for sound recognition and positioning provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0076] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0077] In the description of the present invention, the terms "inside", "outside", "longitudinal", "lateral", "upper", "lower", "top", "bottom" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and do not require that the present invention must be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on the present invention.
[0078] The terms "first", "second", etc. in the present invention are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0079] In the present application, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium.
[0080] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0081] Embodiment 1:
[0082] In order to solve the above problems, Figure 1 As shown, an embodiment of the present invention provides a method for sound recognition and positioning, including:
[0083] Step 10: Extract FBank features and LPC features of at least three audio data.
[0084] Among them, the filter bank feature (FilterBank, abbreviated as FBank) feature processes the audio in a way similar to the nonlinear response of the human ear to the sound spectrum, thereby improving the performance of speech recognition. The linear predictive coding (Linearpredictive coding, abbreviated as LPC) feature represents the spectral envelope of the digital speech signal in a compressed form based on the information of the linear prediction model, which can provide very accurate speech parameter prediction.
[0085] The audio data is an audio signal obtained by performing speech enhancement on the corresponding separated audio signal.
[0086] The sound source device sends out a separated audio signal to extract audio data of at least three sound source devices, and extracts MFCC features and LPC features after performing speech enhancement processing.
[0087] The original audio signal collected from the environment is often mixed with sounds emitted by multiple sound source devices in the environment, as well as environmental noise (such as human voices or decoration noise). Therefore, in an embodiment of the present invention, the audio data emitted by a certain sound source device is separated from the original audio signal as the separated audio signal of the sound source device; the method of separating the separated audio signals of each sound source device from the original audio signal is implemented by those skilled in the art with reference to the prior art. Since there is often too much noise in the separated audio signal, signal distortion during processing will lead to low recognition and positioning accuracy. Therefore, the embodiment of the present invention performs speech enhancement on the separated audio signal, and uses the enhanced audio signal as audio data for sound recognition and positioning; the speech enhancement process will be specifically described below.
[0088] The following is an example of a specific use scenario of the method for voice recognition and positioning according to an embodiment of the present invention:
[0089] like Figure 2 As shown, there are four devices in the indoor environment: a user's mobile phone, a television, a microwave oven and a washing machine, among which the television, the microwave oven and the washing machine are sound source devices that can emit audio signals, and the user's mobile phone is a user device that needs to be located.
[0090] The embodiments of the present invention respectively convert the formats of original audio signals emitted by a television, a microwave oven and a washing machine to obtain audio data in a specified unified format, and then respectively extract FBank features and LPC features of the three audio data.
[0091] The device category to which the sound source device belongs is the sound source category. In an optional embodiment, the sound source category may be the category of the sound source device; for example, the sound source category corresponding to the audio data of a television is a television, and the sound source category corresponding to the audio data of a microwave oven is a microwave oven.
[0092] The distance between the user device and the sound source device is the sound source distance. The distance between the user's mobile phone and the TV is sound source distance 1; the distance between the user's mobile phone and the washing machine is sound source distance 2; the distance between the user's mobile phone and the microwave oven is sound source distance 3.
[0093] Step 20: Obtain a multi-model splicing feature of the audio data based on the FBank feature and the LPC feature of each audio data; model the patterns related to the sound source category and the sound source distance in the multi-model splicing feature to obtain a joint feature representation; wherein the sound source distance is the distance between the sound source device and the user device, and the user device is used to collect audio data of the at least three sound source devices.
[0094] An embodiment of the present invention provides a joint classification and regression network. Based on the network structure of the joint classification and regression network, a corresponding joint classification and regression model is trained. Then, the joint classification and regression model can be used to realize the prediction of sound source category and sound source distance based on FBank features and LPC features, which will be described in detail below.
[0095] Step 30: predict the sound source category to which the audio data belongs based on the joint feature representation; predict the sound source distance of the audio data based on the joint feature representation.
[0096] Step 40: Determine the location of the user equipment according to the sound source categories and sound source distances of at least three audio data.
[0097] In an optional embodiment, three-dimensional audio spheres can be constructed respectively with the positions of the sound source devices corresponding to at least three types of audio data as the center and at least three sound source distances output by the joint classification regression model as the radius. The positions of the corresponding sound source devices are collected; the intersections of different three-dimensional audio spheres are determined, and the plane coordinates of the sound source device and the plane coordinates of the user device are assumed, and finally the coordinates of the user device are solved based on the distance between the sound source device and the user device.
[0098] like Figure 3As shown, for example, the sound source distance 1 between the user's mobile phone and the TV is used as the radius of the three-dimensional audio sphere, and the position of the TV is used as the center of the three-dimensional audio sphere to construct the corresponding three-dimensional audio sphere; the sound source distance 2 between the user's mobile phone and the washing machine is used as the radius of the three-dimensional audio sphere, and the position of the washing machine is used as the center of the three-dimensional audio sphere to construct the corresponding three-dimensional audio sphere; the sound source distance 3 between the user's mobile phone and the microwave oven is used as the radius of the three-dimensional audio sphere, and the position of the microwave oven is used as the center of the three-dimensional audio sphere to construct the corresponding three-dimensional audio sphere. Then, based on the intersection of the three three-dimensional audio spheres, the plane position of the user's mobile phone is determined.
[0099] The present invention extracts FBank features and LPC features of at least three audio data, generates multi-model splicing features of audio data based on them, and models the patterns related to the sound source category and sound source distance therein to obtain a joint feature representation, thereby respectively realizing the prediction of the sound source category and sound source distance to which the audio data belongs; finally, according to the sound source category and sound source distance of at least three audio data, at least three three-dimensional audio spheres are constructed, and the intersection of the constructed multiple three-dimensional audio spheres is determined as the position of the user device. Among them, the present invention only uses FBank features and LPC features, which greatly reduces the calculation complexity and cost, and has high practicality; by jointly analyzing the sound source category and sound source distance, the positioning effect is greatly improved in combination with the time series features; only at least three audio data need to be obtained to realize the positioning of the user device, and the positioning accuracy is high.
[0100] Since there is often too much noise in the directly separated audio signal, in order to avoid signal distortion during processing, resulting in low recognition and positioning accuracy, such as Figure 4 As shown, before step 10, it also includes:
[0101] Step 101: obtaining singular values corresponding to the original audio signal through the SVD algorithm; obtaining an estimate of the number of sound sources based on the singular values and the corresponding attenuation rates; and processing the original audio signal through the FastICA algorithm according to the estimate of the number of sound sources to obtain at least one independent sound source signal.
[0102] Among them, the singular value decomposition (SVD) algorithm decomposes the matrix of the original audio signal into the product of three specific matrices, that is, the original matrix = left singular matrix × singular value diagonal matrix × transpose of the right singular matrix, and then decomposes the original audio signal into multiple frequency components. By analyzing and processing these components, a clear sound signal is finally obtained.
[0103] In the embodiment of the present invention, the sound source of the original audio signal is relatively complex (for example, the original audio signal is collected by multiple microphones), and the singular values of each original audio signal are calculated, and these singular values are arranged from large to small. The interval between multiple singular values is the attenuation amount, and the attenuation speed between the specified singular values can be obtained according to the attenuation amount. An attenuation speed threshold is set, and for the original audio signal, it is truncated at the position where the attenuation speed exceeds the attenuation speed threshold, and the number of singular values before the preset number is taken as the estimate of the number of sound sources; wherein, the attenuation speed threshold and the preset number are selected by those skilled in the art according to the specific usage scenario.
[0104] Then, the estimated number of sound sources is input into the FastICA (Fast Independent Component Analysis) algorithm, which maximizes the non-Gaussianity of the signal and finds the independent components in the original sound source signal, that is, separates at least one independent sound source signal.
[0105] Step 102: using a self-supervised learning method, using a WavLM model to encode the independent sound source signal and extract original continuous features therein; clustering the original continuous features using a K-Means clustering algorithm to convert the original continuous features into initial discrete representations.
[0106] The embodiment of the present invention uses the pre-trained WavLM model to encode the input noisy audio data and extract its continuous features. Among them, self-supervised learning aims to mine the representation characteristics of the data itself as supervisory information for unlabeled data by designing auxiliary tasks (Proxy tasks), thereby improving the feature extraction ability of the model.
[0107] Step 103: using a Transformer-based Decoder-Only structural language model to extract context information from the initial discrete representation, modeling the initial discrete representation based on the context information, and predicting a target discrete representation after speech enhancement.
[0108] This model draws on the architecture of the GPT series, predicts the target discrete representation in a non-autoregressive way, and further improves the effect of speech enhancement by capturing the contextual information in the initial discrete representation.
[0109] Step 104: convert the target discrete representation back to the target continuous feature through a deserializer; and restore the target continuous feature to the sound source sample to be judged through the HiFi-GAN model.
[0110] The de-tokenizer converts the discrete representation back to the original continuous features. The High-Fidelity Generative Adversarial Networks (HiFi-GAN) model is a high-quality audio generation model, which is used by the embodiment of the present invention to restore the continuous features to clear and natural speech signals (i.e., the sound source samples to be determined).
[0111] Step 105: Determine the cosine distance between the sound source sample to be determined and the preset sound source sample in the sound source library, and obtain the similarity between the sound source sample to be determined and the preset sound source sample based on the cosine distance; when the similarity is greater than a similarity threshold, determine the sound source sample to be determined as audio data.
[0112] In an optional embodiment, a sound source sample library is defined in advance for collecting preset sound source samples; the preset sound source samples carry coordinates for marking the sound source devices; and the environmental noise does not carry coordinates.
[0113] In the embodiment of the present invention, cosine similarity calculation is performed between all preset sound source samples carrying coordinates and the sound source sample to be judged, and whether the sound source sample to be judged is in the sound source sample library is determined according to the similarity; if so, its FBank feature and LPC feature are extracted; if not, it is identified as environmental noise and filtered out.
[0114] In step 10, in an optional embodiment, the FBank feature can be obtained by performing a series of processing such as pre-emphasis, framing, windowing, short-time Fourier transform and Mel filtering on the audio data.
[0115] In an optional embodiment, the process of extracting LPC features from each audio data is as follows:
[0116] The amplitude of the high frequency component in the audio data is reduced to obtain pre-emphasized data; the pre-emphasized data is divided into a plurality of frames according to a preset division rule, and each frame is subjected to windowing processing to obtain frame data.
[0117] Among them, the preset division rule is selected by technical personnel in this field according to the specific usage scenario and is not limited here; in an optional embodiment, the division can be performed in a manner that each frame contains 20ms to 30ms of audio data and there is a certain overlap between adjacent frames.
[0118] Perform LPC model fitting on each frame data to obtain LPC coefficients; convert the LPC coefficients into LPCC coefficients; perform normalization processing on the LPCC coefficients to obtain processed coefficients; and concatenate the processed coefficients of all frame data to obtain LPCC features of the audio data.
[0119] The audio data can be pre-emphasized to reduce the amplitude of the high-frequency component; the audio data is divided into several frames, each frame usually contains 20ms to 30ms of audio data, and there is a certain overlap between adjacent frames; each frame is windowed; an LPC model is fitted to each frame after the windowing process to obtain the LPC coefficient; the obtained LPC coefficient is converted into an LPCC coefficient; the LPCC coefficient of each frame after the windowing process is normalized, and then the LPCC coefficients of all the windowed frames are spliced together to obtain the LPCC features of the entire audio data.
[0120] The following is a description of the process of determining the location of a user device:
[0121] In an optional embodiment, if Figure 5 As shown, in step 401a, for each of the at least three sound source devices, the first position coordinate of the sound source device is used as the center of the corresponding three-dimensional audio sphere, and the sound source distance corresponding to the sound source device is used as the radius of the three-dimensional audio sphere to construct the three-dimensional audio sphere. The first position coordinate is the plane coordinate of the sound source device; for example, the plane coordinate of the first sound source device is ( , ), the plane coordinates of the second sound source device are ( , ), the plane coordinates of the third sound source device are ( , ).
[0122] The intersection of each three-dimensional audio sphere is determined as the second position coordinate of the user device to determine the position of the user device. The second position coordinate is the plane coordinate (x, y) of the user device. Each three-dimensional audio sphere refers to all constructed three-dimensional audio spheres. The difference between the user's horizontal coordinate to be solved and the sound source horizontal coordinate of each sound source device is used as the corresponding horizontal distance; the difference between the user's vertical coordinate to be solved and the sound source vertical coordinate of each sound source device is used as the corresponding vertical distance; construct the spatial position relationship between each horizontal distance and the corresponding vertical distance, as well as the corresponding sound source distance. For example, the horizontal distance corresponding to the first sound source device is (x- ); where x is the user's horizontal coordinate, is the horizontal coordinate of the sound source. The horizontal distance corresponding to the second sound source device is (x- ). The horizontal distance corresponding to the third sound source device is (x- ). The vertical distance corresponding to the first sound source device is (y- ); where y is the user's ordinate, is the vertical coordinate of the sound source. The vertical distance corresponding to the second sound source device is (y- ). The vertical distance corresponding to the third sound source device is (y- ). The sound source distance of each sound source device is obtained by prediction; for example, the sound source distance corresponding to the first sound source device is , the sound source distance corresponding to the second sound source device is , the sound source distance corresponding to the third sound source device is The spatial position relationship formula is a set of equations consisting of expressions of at least three horizontal distance quantities, vertical distance quantities and corresponding sound source distances.
[0123] That is, in the case of audio data from three sound source devices, the spatial position relationship is obtained based on the intersection of the three three-dimensional audio spheres as follows:
[0124]
[0125] On this basis, when there are four sound source devices, in an optional embodiment, the spatial position relationship expression can be:
[0126]
[0127] Among them, the plane coordinates of the fourth sound source device are ( , ), the sound source distance corresponding to the fourth sound source device is When there are more than four sound source devices, the spatial position relationship expression is similar.
[0128] In step 402a, the intersection points of the three-dimensional audio spheres are determined as the second position coordinates of the user equipment to determine the position of the user equipment.
[0129] Solve the spatial position relationship equation, and set the initial guess value of the equation to be solved in the spatial position relationship equation based on the solution of the preset equation in the spatial position relationship equation , according to the initial guess value Solve the equation to be solved to obtain the user horizontal coordinate and the user vertical coordinate to obtain the second position coordinate; wherein, is the user horizontal coordinate of the equation to be solved, is the user ordinate of the equation to be solved; wherein the preset equations may be the first two equations in the set of equations to be solved.
[0130] When using multiple sound source devices for positioning, due to measurement errors such as reverberation or the location of the sound source, when the number of equations exceeds the number of unknowns, there may be no solution or multiple solutions. Due to reverberation and measurement errors, the location coordinates of the receiver ( , ) and the distance from the sound source It may be inaccurate, resulting in the system of equations having no exact solution or having multiple sets of approximate solutions.
[0131] The embodiment of the present invention adopts the following method to obtain more accurate user equipment location coordinates, and the solution idea is: each time the solution of the preset equation is substituted into the equation to be solved, the equation to be solved is solved, and then the system of equations to be solved is iteratively solved. Specifically, the solving of the spatial position relationship equation to obtain the user horizontal coordinate and the user vertical coordinate includes:
[0132] The measurement error of the i-th sound source device is expressed as the first formula; wherein the expression of the first formula is: = , is the sound source horizontal coordinate of the i-th sound source device, is the sound source ordinate of the i-th sound source device, is the sound source distance of the i-th sound source device.
[0133] The first formulas of all sound source devices are combined into a measurement vector; the expression of the measurement error vector is: ;in, is the first formula for the i-th sound source device.
[0134] For each sound source device, calculate the measurement error vector For the current guess value The Jacobian matrix of ; wherein the expression of the Jacobian matrix is: , the i-th sound source device refers to a sound source device among all the sound source devices currently acquired; the parameter update step length is determined according to the Jacobian matrix; the expression of the parameter update step length is: ;
[0135] Based on the initial guess value and the parameter update step size, iteratively updating the corresponding current guess value ; Among them, the kth sound source device refers to a sound source device among all the sound source devices currently obtained; the updated current guess value The expression is: = + .
[0136] When the objective function S When the change in is less than the first preset threshold, the norm of the parameter update step is less than the second preset threshold, or the maximum number of iterations is reached, the iteration is stopped to obtain the user horizontal coordinate and the corresponding user vertical coordinate, that is, the final coordinate estimation result of the user device is obtained at this time.
[0137] In another optional embodiment, if Figure 6 As shown, the step 40 includes:
[0138] In step 401b, for each of the at least three sound source devices, the first position coordinate of the sound source device is used as the center of the corresponding three-dimensional audio circle, and the sound source distance corresponding to the sound source device is used as the radius of the three-dimensional audio circle to construct the three-dimensional audio circle.
[0139] In step 402b, the differences between the user's horizontal coordinate to be solved and the sound source horizontal coordinate of each sound source device are taken as the corresponding horizontal distance; the differences between the user's vertical coordinate to be solved and the sound source vertical coordinate of each sound source device are taken as the corresponding vertical distance; and the horizontal distance and vertical distance corresponding to each of the three-dimensional audio circles and the spatial position relationship between the corresponding sound source distances are constructed.
[0140] In step 403b, the spatial position relationship equation is solved to obtain the user horizontal coordinate and the user vertical coordinate to determine the position of the user device (i.e., determine the second position coordinate of the user device). The second position coordinate of the embodiment of the present invention is: the final coordinate estimation result of the user device obtained by prediction based on the audio data.
[0141] The method for determining the second position coordinates is as follows: Same as the above method, set the initial particle swarm guess value according to the solution of the equation , the objective function is defined as the residual sum of squares of all measurement errors; where the expression of the objective function is: , is the sound source horizontal coordinate of the i-th sound source device, is the sound source ordinate of the i-th sound source device, is the sound source distance of the i-th sound source device; sets the maximum number of iterations T and inertia weight and two learning factors: learning factor and learning factor , randomly generate a random seed and random seed .
[0142] Each two equations in the spatial position relationship are regarded as a system of equations, and the solution of the system of equations is set as the initial position of the particle; where the solution of each system of equations is ( ), ; The initial velocity of particles is 0;
[0143] Iteratively calculating the objective function value of each particle according to the objective function based on the initial position and the initial velocity;
[0144] If the objective function value of the particle's new position is greater than the objective function value of its individual optimal position, the individual optimal position is updated using the new position; if the objective function value of the new position is greater than the objective function value of the global optimal position, the global optimal position is updated using the new position; that is, for each iteration t, the fitness value of each particle is calculated. If the objective function value of the particle's new position is better than its individual optimal, the individual optimal position of the particle is updated. If the objective function value of the particle's new position is better than the global optimal position, then the global optimal position is updated. For the i-th particle solution, update its velocity using the following formula:
[0145] +
[0146] +
[0147]
[0148]
[0149] in, For the The vertical velocity of a particle is For the The velocity of a particle in the horizontal direction, and the inertia weight is , is a learning factor, is another learning factor, is a random seed, is another random seed; The new position of the particle is ( , ), No. The individual optimal position of a particle is ( , ), No. The global optimal position of a particle is ( , ).
[0150] When the maximum number of iterations T is reached or the change in the objective function value of the global optimal position is less than a third preset threshold, the iteration is terminated, and the final coordinate estimation result of the user device is output; the third preset threshold is selected by technical personnel in this field according to the specific usage scenario.
[0151] The second position coordinates predicted by the embodiment of the present invention are continuously moved to a new, known and determined position ( ), ), ...; and at the same time, the particle swarm is iterated and the calculated result is subtracted from the position coordinates determined by the previous iteration to obtain the relative error ( ), )..., average all the calculated relative errors to get the system error ( )
[0152] In the process of particle swarm iterative calculation, when the position of the second position coordinate predicted by the embodiment of the present invention is at a certain unknown and unresolved coordinate, the above-mentioned particle swarm algorithm is used to calculate the optimization result of the N-variable equation group at this time ( , ), and then add the systematic error ( ) and recalculate the objective function to obtain the corrected optimal result, and then output the final coordinate estimation result of the user device predicted based on the audio data.
[0153] Embodiment 2:
[0154] In order to illustrate the joint classification regression model of the embodiment of the present invention, Figure 7 As shown, the step 20 includes:
[0155] Step 201: Adaptively average pool the FBank features to obtain compressed features.
[0156] like Figure 8 The figure shows the network structure of the joint classification regression network of an embodiment of the present invention. In the process of training a corresponding joint classification regression model based on the network structure, a set of inputs for the training model are: 1. FBank features and LPC features of an audio data; 2. The sound source distance of the audio data (i.e., a sound source distance label); 3. The sound source category of the audio data (i.e., a sound source category label).
[0157] Since the high-dimensional FBank features often contain redundant information, in order to reduce the amount of calculation and improve the generalization ability of the model, such as Figure 8 As shown, an embodiment of the present invention compresses the extracted FBank features into data in the time dimension through an adaptive average pooling layer; in an optional embodiment, the adaptive average pooling layer can be implemented using AdaptiveAvgPool1d to reduce the input FBank features to a specified size.
[0158] Step 202: stack the compressed features and the LPC features in the channel dimension according to a preset splicing method to obtain the multi-model splicing features.
[0159] The preset splicing method is selected by those skilled in the art according to the specific usage scenario; in an optional embodiment, feature splicing can be performed according to the following method:
[0160] The compressed feature contains n sub-features: [f1_A, f2_A, ..., fn_A]; the LPC feature contains m sub-features: [f1_B, f2_B, ..., fm_B]; by splicing on the feature dimension, a multi-model splicing feature is obtained, which contains n+m sub-features: [f1_A, f2_A, ..., fn_A, f1_B, f2_B, ..., fm_B].
[0161] Step 203: Input the multi-model splicing features into multiple cascaded audio sequence modeling modules, wherein the input features of the first audio sequence modeling module are the multi-model splicing features; the input features of other audio sequence modeling modules are the output features of the previous audio sequence modeling module, and the joint feature representation is obtained from the last audio sequence modeling module.
[0162] The last audio sequence modeling module refers to the last one of the multiple sequentially connected audio sequence modeling modules. The multi-model concatenation feature is used as the input feature of the first audio sequence modeling module among the multiple sequentially connected audio sequence modeling modules, the first audio sequence modeling module processes the input feature to generate the corresponding output feature, and then the output feature is used as the input feature of the second audio sequence modeling module among the multiple sequentially connected audio sequence modeling modules, and so on.
[0163] Among them, the execution logic of each audio sequence modeling module is: learn the dependency of sound source category and / or sound source distance from the input features, obtain output features that characterize the dependency, and input the output features to the next audio sequence modeling module so that the next audio sequence modeling module can update the currently learned dependency.
[0164] Among them, one audio sequence modeling module corresponds to one input feature and one output feature, and each audio sequence modeling module generates its own output feature based on its own input feature. The dependency relationship refers to the relationship between the multi-model concatenation feature and the sound source category or sound source distance, and the relationship between the multi-model concatenation feature and the sound source category and sound source distance; the dependency relationship is specifically determined by the network structure adopted by the audio sequence modeling module and the features that are actually implicit in the input features and can be learned.
[0165] For FBank features and LPC features, in order to more efficiently and effectively capture relevant complex patterns and relationships across long sequences, an embodiment of the present invention uses a corresponding network structure to perform sequential modeling on them: each audio sequence modeling module sequentially processes multi-model splicing features, and the output of each audio sequence modeling module serves as the input of the next audio sequence modeling module.
[0166] The FBank features and LPC features of the same audio data are obtained based on different principles and audio representation models. The embodiment of the present invention uses a plurality of sequentially connected audio sequence modeling modules to analyze and extract features related to the sound source category and the sound source distance from the multi-model splicing features after splicing the two. Specifically, the first audio sequence modeling module learns the output features used to predict the sound source category and the sound source distance from the multi-model splicing features, and compresses the input features by selecting features; for each of the remaining audio sequence modeling modules, each time referring to the summary of the dependency relationship obtained by all the audio sequence modeling modules before it, selectively pays attention to the features that must be paid attention to, filters the features that can be ignored, that is, updates the dependency relationship extracted by the previous audio sequence modeling module, and generates the compressed output features of this module. Finally, the joint feature representation learned by multiple sequentially connected audio sequence modeling modules not only contains features related to the sound source category and the sound source distance, but also is obtained based on the sequential relationship between each frame in the audio data, thereby further enhancing the feature recognition ability of the joint classification regression model of the embodiment of the present invention.
[0167] The feature extraction operation in each audio sequence modeling module is described below. Fig. 9 As shown, the dependency relationship of the sound source category and / or the sound source distance is learned from the input features, and the output features representing the dependency relationship are obtained, including:
[0168] Step 2031: Perform a linear transformation on the input feature to expand the dimension of the input feature to obtain an increased-dimensional feature representation.
[0169] Step 2032: Use a first activation function to learn the nonlinear features of the dimension-increased feature representation to obtain a first nonlinear feature representation; the expression of the first activation function is: ; Wherein, a is the feature of the first activation function input.
[0170] The first activation function may also be selected by those skilled in the art according to a specific usage scenario.
[0171] Step 2033: performing a convolution operation on the increased-dimensional feature representation to extract local features of the increased-dimensional feature representation to obtain a local feature representation; using a first activation function to learn the nonlinear features of the local feature representation to obtain a second nonlinear feature representation; assigning corresponding attention weights to each feature in the second nonlinear feature representation, compressing the second nonlinear feature representation based on the attention weights, and extracting a third nonlinear feature representation.
[0172] Before performing feature compression based on the attention mechanism, the embodiment of the present invention first performs a convolution operation to extract long-distance dependencies in the time dimension from the multi-model splicing features of the audio data based on the corresponding context or relationship when processing the features, and selectively scans the features each time according to the dependencies extracted by the current audio sequence modeling module to selectively compress the corresponding information therein.
[0173] Step 2034: nonlinearly connect the first nonlinear feature representation and the third nonlinear feature representation to obtain output features.
[0174] The nonlinear connection can be implemented using an activation function, or by multiplying the features in the first nonlinear feature representation and the third nonlinear feature representation one by one.
[0175] like Fig.10 As shown, each audio sequence modeling module of the embodiment of the present invention first uses an input transformation layer to expand the dimension of the input multi-model concatenated features to obtain a corresponding increased-dimensional feature representation.
[0176] It is divided into two paths to process the increased-dimensional feature representation; one path is: use the first activation function to learn its nonlinear features to obtain the first nonlinear feature representation; the other path is: the convolution layer first performs a convolution operation on the increased-dimensional feature representation, extracts the local feature representation, and then uses the first activation function to learn the second nonlinear feature representation from the local feature representation; then the feature selection module is used to obtain the third nonlinear feature representation corresponding to the second nonlinear feature representation based on the attention mechanism.
[0177] Next, the first nonlinear feature representation extracted from the two paths is nonlinearly connected with the third nonlinear feature representation, and finally the features obtained after the nonlinear connection are linearly mapped to the corresponding joint feature representation through the output transformation layer.
[0178] The embodiment of the present invention processes FBank features and LPC features based on multiple sequentially connected audio sequence modeling modules, and then needs to complete the prediction task of sound source category and sound source distance based on the obtained joint feature representation. Specifically, Fig.11 As shown, in the step 30, predicting the sound source category to which the audio data belongs based on the joint feature representation includes:
[0179] Step 301a: Linearly map the joint feature representation into a two-dimensional feature representation.
[0180] like Figure 8 As shown, since multiple audio sequence modeling modules connected in sequence output the shape of the data, that is, three-dimensional data, the embodiment of the present invention determines the sound source category by obtaining a category probability distribution of the sound source category. This category probability distribution belongs to two-dimensional data, so feature dimensionality reduction needs to be performed first.
[0181] Step 302a: Use a second activation function to learn the nonlinear features of the two-dimensional feature representation to obtain a classification feature representation; the expression of the second activation function is: ; Wherein, b is the feature input to the second activation function.
[0182] Among them, the second activation function can also be selected by a person skilled in the art according to a specific usage scenario. Since a probability distribution needs to be obtained, the embodiment of the present invention needs to use an activation function with a relatively higher output range. The value range of the second activation function of the embodiment of the present invention is [-1, 1], which has a strong characterization ability and a larger operating space in the calculation of probability distribution.
[0183] Step 303a: Linearly map the classification feature representation into a category probability distribution.
[0184] Step 304a: Determine the sound source category from the category probability distribution according to a preset selection rule.
[0185] Among them, the preset selection rules are selected by technical personnel in this field according to the specific usage scenario and are not limited here; in an optional embodiment, a category with a large probability can be selected from the category probability distribution as the sound source category of the corresponding sound source device predicted by the embodiment of the present invention.
[0186] The process of obtaining the category probability distribution completes a classification task. After many experiments, the embodiment of the present invention determines that the joint feature representation output by the audio sequence modeling module performs well in the classification task performed by the category prediction module, so only one activation function is used to extract the sound source category information contained in the joint feature representation, and linearly map it to the category probability distribution through the first output layer.
[0187] The following is an explanation of the process of predicting the sound source distance. Fig.12 As shown, in the step 30, predicting the sound source distance of the audio data according to the joint feature representation includes:
[0188] Step 301b: Linearly map the joint feature representation into a one-dimensional feature representation.
[0189] like Figure 8 As shown, since multiple audio sequence modeling modules connected in sequence output the shape of the data, that is, three-dimensional data, but the embodiment of the present invention needs to obtain a numerical value of the sound source distance, that is, one-dimensional data, it is necessary to perform feature dimensionality reduction first.
[0190] Step 302b: Use a third activation function to learn the nonlinear features of the one-dimensional feature representation to obtain an intermediate regression feature representation; the expression of the third activation function is: ; Wherein, c is the feature of the third activation function input.
[0191] Among them, the third activation function can also be selected by those skilled in the art according to the specific usage scenario. Since a value of the sound source distance needs to be obtained, the output range of the activation function has little effect on the obtained value, and the third activation function with a value range between [c, 0] is used to perform nonlinear processing on the one-dimensional feature representation.
[0192] Step 303b: Use the third activation function to learn the nonlinear features of the intermediate regression feature representation to obtain the target regression feature representation.
[0193] The process of obtaining the sound source distance completes a regression task. After many experiments, the embodiment of the present invention determines that the joint feature representation output by the audio sequence modeling module performs slightly worse than the classification task of category prediction in the regression task performed by the distance prediction module. Therefore, under the premise of balancing the computational cost and prediction accuracy, two activation functions are used to increase the nonlinear ability and make the model more robust.
[0194] Step 304b: Linearly map the target regression feature representation to the sound source distance.
[0195] Finally, the sound source distance is output through the second output layer.
[0196] It should be noted that the methods not described in detail in the embodiments of the present invention are well known to those skilled in the art and can be implemented with reference to the prior art. They will not be expanded in the subsequent embodiments and should not be regarded as unclear aspects of the present invention.
[0197] Embodiment 3:
[0198] like Fig.13 , which is a schematic diagram of the architecture of a device for voice recognition and positioning according to an embodiment of the present invention. The device for voice recognition and positioning according to this embodiment includes one or more processors 21 and a memory 22. Fig.13 A processor 21 is taken as an example.
[0199] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.
[0200] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for sound recognition and positioning in this embodiment. The processor 21 executes the method for sound recognition and positioning by running the non-volatile software programs and instructions stored in the memory 22.
[0201] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0202] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the method for sound recognition and positioning in the above embodiment is executed, for example, the method described above is executed. Figure 1 , Figure 4-Figure 7 , Fig. 9 and Figure 11-Figure 12 The steps shown.
[0203] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Fig.13 A processor 21 can enable the above one or more processors to execute the method for sound recognition and positioning in the specific embodiment of the present invention, for example, to execute the above described Figure 1 , Figure 4-Figure 7 , Fig. 9 and Figure 11-Figure 12 The steps shown can also be implemented Fig.13 The modules and units described above; or the method for sound recognition and positioning in a specific embodiment of the present invention, for example, the method described above Figure 1 , Figure 4-Figure 7 , Fig. 9 and Figure 11-Figure 12 The steps shown can also be implemented Fig.13 The various modules and units described.
[0204] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.
[0205] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0206] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for sound recognition and positioning, characterized in that: include: Extract FBank features and LPC features of at least three audio data; According to the FBank feature and the LPC feature of each audio data, a multi-model splicing feature of the audio data is obtained; a pattern related to the sound source category and the sound source distance in the multi-model splicing feature is modeled to obtain a joint feature representation; wherein the sound source distance is the distance between the sound source device and the user device, and the user device is used to collect audio data of the at least three sound source devices; Predicting the sound source category to which the audio data belongs based on the joint feature representation; predicting the sound source distance of the audio data based on the joint feature representation; Determine the location of the user device according to the sound source category and the sound source distance of at least three audio data; Among them, predicting the sound source category to which the audio data belongs according to the joint feature representation includes: linearly mapping the joint feature representation into a two-dimensional feature representation; using a second activation function to learn the nonlinear features of the two-dimensional feature representation to obtain a classification feature representation; the expression of the second activation function is: ; Wherein, b is the feature input to the second activation function; the classification feature representation is linearly mapped into a category probability distribution; according to a preset selection rule, the sound source category is determined from the category probability distribution.
2. The method for voice recognition and positioning according to claim 1, characterized in that: According to the FBank feature and LPC feature of each audio data, a multi-model splicing feature of the audio data is obtained; Modeling the patterns related to the sound source category and the sound source distance in the multi-model concatenation features to obtain a joint feature representation including: Adaptively average pooling the FBank features to obtain compressed features; According to a preset splicing method, the compressed features and the LPC features are stacked in the channel dimension to obtain the multi-model splicing features; Inputting the multi-model concatenation feature into a plurality of cascaded audio sequence modeling modules, wherein the input feature of the first audio sequence modeling module is the multi-model concatenation feature; the input feature of the other audio sequence modeling modules is the output feature of the previous audio sequence modeling module, and the joint feature representation is obtained from the last audio sequence modeling module; Among them, the execution logic of each audio sequence modeling module is: learn the dependency of sound source category and / or sound source distance from the input features, obtain output features that characterize the dependency, and input the output features to the next audio sequence modeling module so that the next audio sequence modeling module can update the currently learned dependency.
3. The method for voice recognition and positioning according to claim 2, characterized in that: The step of learning the dependency relationship between the sound source category and / or the sound source distance from the input features and obtaining the output features representing the dependency relationship includes: Performing a linear transformation on the input feature to expand the dimension of the input feature to obtain an increased-dimensional feature representation; The nonlinear features of the dimension-increased feature representation are learned using a first activation function to obtain a first nonlinear feature representation; the expression of the first activation function is: ; Wherein, a is the feature input into the first activation function; Performing a convolution operation on the increased-dimensional feature representation to extract local features of the increased-dimensional feature representation to obtain a local feature representation; using a first activation function to learn nonlinear features of the local feature representation to obtain a second nonlinear feature representation; assigning a corresponding attention weight to each feature in the second nonlinear feature representation, compressing the second nonlinear feature representation based on the attention weight, and extracting a third nonlinear feature representation; The first nonlinear feature representation and the third nonlinear feature representation are nonlinearly connected to obtain output features.
4. The method for voice recognition and positioning according to claim 1, characterized in that: The predicting the sound source distance of the audio data according to the joint feature representation comprises: Linearly mapping the joint feature representation into a one-dimensional feature representation; The nonlinear features of the one-dimensional feature representation are learned using a third activation function to obtain an intermediate regression feature representation; the expression of the third activation function is: ; Wherein, c is the feature of the third activation function input; Using a third activation function to learn the nonlinear features of the intermediate regression feature representation to obtain a target regression feature representation; The target regression feature representation is linearly mapped to the sound source distance.
5. The method for voice recognition and positioning according to claim 1, characterized in that: The determining the position of the user equipment according to the sound source category and the sound source distance of at least three audio data comprises: For each of the at least three sound source devices, taking the first position coordinate of the sound source device as the center of the corresponding three-dimensional audio circle, taking the sound source distance corresponding to the sound source device as the radius of the three-dimensional audio circle, and constructing the three-dimensional audio circle; The difference between the user's horizontal coordinate to be solved and the sound source horizontal coordinate of each sound source device is used as the corresponding horizontal distance; the difference between the user's vertical coordinate to be solved and the sound source vertical coordinate of each sound source device is used as the corresponding vertical distance; and a spatial position relationship between the horizontal distance and the vertical distance corresponding to each three-dimensional audio circle and the corresponding sound source distance is constructed; The spatial position relationship equation is solved to obtain the user horizontal coordinate and the user vertical coordinate to determine the position of the user equipment.
6. The method for voice recognition and positioning according to claim 5, characterized in that: Solving the spatial position relationship equation to obtain the user horizontal coordinate and the user vertical coordinate includes: Each two equations in the spatial position relationship are regarded as a system of equations, and the solution of the system of equations is set as the initial position of the particle; where the solution of each system of equations is ( ), ; The initial velocity of particles is 0; Based on the initial position and the initial velocity, the objective function value of each particle is iteratively calculated according to the objective function; wherein the expression of the objective function is: , is the sound source horizontal coordinate of the i-th sound source device, is the sound source ordinate of the i-th sound source device, is the sound source distance of the i-th sound source device; If the objective function value of the particle's new position is greater than the objective function value of its individual optimal position, the individual optimal position is updated using the new position; if the objective function value of the new position is greater than the objective function value of the global optimal position, the global optimal position is updated using the new position; in each iteration, for the For each particle, update its velocity and position using the following formula: + + in, For the The vertical velocity of a particle is For the The velocity of a particle in the horizontal direction, and the inertia weight is , is a learning factor, is another learning factor, is a random seed, is another random seed; The new position of the particle is ( , ), No. The individual optimal position of a particle is ( , ), No. The global optimal position of a particle is ( , ); When the maximum number of iterations is reached or the change in the objective function value of the global optimal position is less than the convergence criterion, the iteration is terminated; the horizontal coordinate of the global optimal position after the termination of the iteration is used as the user horizontal coordinate, and the vertical coordinate of the global optimal position after the termination of the iteration is used as the user vertical coordinate.
7. The method for voice recognition and positioning according to claim 1, characterized in that: Before extracting the FBank features and LPC features of at least three audio data, performing speech enhancement on the original audio signal to obtain corresponding audio data; The method for sound recognition and positioning also includes: Obtaining the singular value corresponding to the original audio signal by using the SVD algorithm; Obtaining an estimate of the number of sound sources according to the singular values and the corresponding attenuation rates; and processing the original audio signal by using a FastICA algorithm according to the estimate of the number of sound sources to obtain at least one independent sound source signal; The independent sound source signal is encoded using the WavLM model in a self-supervised learning manner to extract the original continuous features therein; the original continuous features are clustered using a K-Means clustering algorithm to convert the original continuous features into initial discrete representations; A Transformer-based Decoder-Only structure language model is used to extract context information from the initial discrete representation, model the initial discrete representation based on the context information, and predict a target discrete representation after speech enhancement; The target discrete representation is converted back to the target continuous feature through a deserializer; the target continuous feature is restored to the sound source sample to be judged through a HiFi-GAN model; Determine the cosine distance between the sound source sample to be determined and a preset sound source sample in a sound source library, and obtain the similarity between the sound source sample to be determined and the preset sound source sample based on the cosine distance; when the similarity is greater than a similarity threshold, determine the sound source sample to be determined as audio data.
8. A device for sound recognition and positioning, characterized in that: The device for sound recognition and positioning includes at least one processor and a memory, and the at least one processor and the memory are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to implement the method for sound recognition and positioning described in any one of claims 1-7.
9. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method for sound recognition and positioning described in any one of claims 1-7.
Citation Information
Patent Citations
Sound event detection and positioning method based on deep learning
CN111443328A
Method and apparatus for determining location of mobile device
US20120142378A1
