Environmental sound recognition method and device based on deep learning composite model

Abstract

The present invention proposes a method and device for environmental sound recognition based on a deep learning composite model, the method comprising: obtaining environmental audio, extracting features from the environmental audio to obtain environmental audio features, inputting the environmental audio features into a pre-trained composite model, the composite model comprising a plurality of general models with different network structures; identifying the general recognition confidence corresponding to each category label through the general model; determining the target weighted confidence of each category label as the classification output result of the general model based on the model weights and the general recognition confidence of the general model; and obtaining the output result of the composite model according to the average of the target weighted confidence. According to the technical solution of the embodiment of the present invention, the confidence of environmental audio recognition can be weighted and fused through a plurality of general models with different network structures, the overfitting risk of classification recognition can be reduced, the generalization ability can be enhanced, and the accuracy of environmental audio recognition can be maintained in long-term use scenarios.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a method and device for environmental sound recognition based on a deep learning composite model. Background Art

[0002] With the development of artificial intelligence technology, deep learning models have begun to be applied to the field of noise pollution control. Environmental workers input the collected environmental sounds into the trained deep learning models, and use the deep learning models to identify various types of sounds from the environmental sounds, providing a data basis for the monitoring and control of urban environmental noise.

[0003] Related technologies mainly use a single model to identify environmental sounds, but the types of urban environmental sounds are complex, and it is difficult for a single model to maintain a good recognition effect for a long time. It is easy to overfit, that is, some categories will have a high recognition rate, while other categories will have a low recognition rate. Moreover, urban environmental noise usually has some new noise categories over time. The use of related technologies requires retraining the model after adding new categories, which not only wastes time and resources, but also may not be able to achieve the recognition effect of the retrained model if the new categories are similar to the existing categories. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a method and device for environmental sound recognition based on a deep learning composite model, which can reduce the overfitting risk of classification recognition, enhance generalization ability, and improve the accuracy of environmental sound recognition.

[0005] In a first aspect, an embodiment of the present invention provides an environmental sound recognition method based on a deep learning composite model, comprising:

[0006] Acquire ambient audio, extract features of the ambient audio to obtain ambient audio features, and input the ambient audio features into a pre-trained composite model, wherein the composite model includes multiple trained general models, the general model is a DCNN network model, each general model has a different network structure, the general model is preset with multiple category labels, and the general model is preset with model weights;

[0007] Traversing the composite model, classifying and identifying the environmental audio features through each general model, and obtaining the classification and identification results of each general model, wherein the classification and identification results include the general identification confidence corresponding to each category label;

[0008] Based on the model weights and the universal recognition confidence of the universal model, the target weighted confidence of each category label is determined, and the target weighted confidence is determined as the classification output result of the universal model;

[0009] The composite recognition result of the composite model is determined according to all the classification output results, wherein the composite recognition result includes the composite confidence of each category label, and the composite confidence is the average of all the target weighted confidences corresponding to the category label.

[0010] According to some embodiments of the present invention, extracting features from ambient audio to obtain ambient audio features includes:

[0011] Preprocessing the ambient audio includes eliminating DC components and normalizing the amplitude;

[0012] Based on Log-Mel, PCEN or MFCCs, the ambient audio features are extracted from the preprocessed ambient audio.

[0013] According to some embodiments of the present invention, before acquiring the ambient audio, the method further includes:

[0014] Obtain a training data set, where the training data set includes multiple sample audio features with pre-labeled category labels;

[0015] The training data set is divided into a plurality of training sample sets and a plurality of test sample sets, wherein the sample audio features of each training sample set are different from each other, the sample audio features of each test sample set are different from each other, and the number of the training sample sets and the test sample sets is the same as the number of the general model;

[0016] Each set of training sample sets and test sample sets is input into a common model, and the model training is completed after data enhancement of the sample audio features of the training sample sets.

[0017] According to some embodiments of the present invention, data enhancement includes at least one of the following:

[0018] Based on a preset mixing coefficient, multiple sample audio features are randomly mixed in pairs;

[0019] Smoothing the category labels of the sample audio features;

[0020] Add Gaussian white noise with random signal-to-noise ratio to the sample audio features;

[0021] Randomly mask the time scale of the sample audio features;

[0022] Randomly mask the frequency scale of the sample audio features.

[0023] According to some embodiments of the present invention, at least one general model is further associated with a special case model set, the special case model set includes at least one special case model, the special case model is a DCNN network model, the network structure of each special case model is different, the special case model is preset with multiple category labels, the category label of the special case model is a true subset of the category label of the general model to which it belongs, the category label of the special case model is preset with a label weight, and based on the model weight and the general confidence of the general model, the target weighted confidence of each category label is determined, including:

[0024] The product of the model weight and the universal recognition confidence is determined as the universal weighted confidence;

[0025] Inputting the environmental audio features into the special case model for classification and recognition, obtaining the special case recognition result output by the feature model, wherein the special case recognition result includes the special case recognition confidence of each category label of the special case model;

[0026] The product of the special case recognition confidence and the corresponding label weight is determined as the feature weighted confidence;

[0027] When the category label is associated with a feature weighted confidence and a universal weighted confidence, the mean of the feature weighted confidence and the universal weighted confidence is determined as the target weighted confidence;

[0028] Alternatively, when there is no associated feature weighted confidence for the category label, the universal weighted confidence is determined as the target weighted confidence.

[0029] According to some embodiments of the present invention, inputting the environmental audio features into the special case model for classification and recognition includes:

[0030] Sort multiple category labels from large to small according to the universal weighted confidence;

[0031] Determine multiple high-confidence labels according to the order of sorting from front to back;

[0032] Traverse the special case model, and when the high confidence label is the category label in the special case model, input the environmental audio features into the special case model for classification and recognition.

[0033] According to some embodiments of the present invention, after determining the composite recognition result of the composite model according to all the classification output results, the method further includes:

[0034] When a new general model is obtained, the new general model is configured into the composite model, wherein the new general model is preset with multiple category labels, and at least one of the category labels of the new general model is different from the category label of the general model of the composite model;

[0035] Alternatively, when a new special case model is obtained, the general model associated with the new special case model is determined, and the new special case model is configured into the composite model, wherein the category labels of the new special case model and the category labels of the special case model in the composite model do not overlap with each other;

[0036] Alternatively, when a new general model is obtained and the new general model is associated with a new special case model, the new general model and the new special case model are configured into a composite model, wherein at least one of the category labels of the new general model is different from the category label of the general model of the composite model, and the category label of the new special case model does not overlap with the category label of the special case model in the composite model.

[0037] In a second aspect, an embodiment of the present invention provides an environmental sound recognition device based on a deep learning composite model, comprising at least one control processor and a memory for communicating with at least one control processor; the memory stores instructions that can be executed by at least one control processor, and the instructions are executed by at least one control processor so that at least one control processor can execute the environmental sound recognition method based on the deep learning composite model as described in the first aspect above.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising an environmental sound recognition device based on a deep learning composite model as described in the second aspect above.

[0039] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the environmental sound recognition method based on the deep learning composite model as described in the first aspect above.

[0040] According to the embodiment of the present invention, the method for environmental sound recognition based on deep learning composite model has at least the following beneficial effects: obtaining environmental audio, extracting features of the environmental audio to obtain environmental audio features, and inputting the environmental audio features into a pre-trained composite model, wherein the composite model includes multiple trained general models, the general model is a DCNN network model, each general model has a different network structure, the general model is preset with multiple category labels, and the general model is preset with model weights; traversing the composite model, classifying and identifying the environmental audio features through each general model, respectively, to obtain classification and recognition results of each general model, and the classification and recognition results include the general recognition confidence corresponding to each category label; based on the model weight and the general recognition confidence of the general model, determining the target weighted confidence of each category label, and determining the target weighted confidence as the classification output result of the general model; determining the composite recognition result of the composite model according to all the classification output results, wherein the composite recognition result includes the composite confidence of each category label, and the composite confidence is the average of all the target weighted confidences corresponding to the category label. According to the technical solution of the embodiment of the present invention, the confidence of environmental audio recognition can be weighted and fused through general models of multiple different network structures, which can reduce the overfitting risk of classification recognition, enhance the generalization ability, and maintain the accuracy of environmental audio recognition in long-term use scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0042] Figure 2 is a flow chart of an environmental sound recognition method based on a deep learning composite model provided by an embodiment of the present invention;

[0043] Figure 3 is a flow chart of a specific example of the present invention;

[0044] Figure 4 It is a structural diagram of an environmental sound recognition device based on a deep learning composite model provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0045] Embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0046] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0047] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0048] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0049] The embodiment of the present invention provides an environmental sound recognition method and device based on a deep learning composite model, wherein the environmental sound recognition method based on the deep learning composite model includes: obtaining environmental audio, extracting features of the environmental audio to obtain environmental audio features, and inputting the environmental audio features into a pre-trained composite model, wherein the composite model includes multiple trained general models, the general model is a DCNN network model, each general model has a different network structure, the general model is preset with multiple category labels, and the general model is preset with model weights; traversing the composite model, classifying and identifying the environmental audio features through each general model, respectively, to obtain classification and recognition results of each general model, and the classification and recognition results include the general recognition confidence corresponding to each category label; based on the model weight and the general recognition confidence of the general model, determining the target weighted confidence of each category label, and determining the target weighted confidence as the classification output result of the general model; determining the composite recognition result of the composite model according to all the classification output results, wherein the composite recognition result includes the composite confidence of each category label, and the composite confidence is the average of all the target weighted confidences corresponding to the category label. According to the technical solution of the embodiment of the present invention, the confidence of environmental audio recognition can be weighted and fused through general models of multiple different network structures, which can reduce the overfitting risk of classification recognition, enhance the generalization ability, and maintain the accuracy of environmental audio recognition in long-term use scenarios.

[0050] Reference Figure 1 , Figure 1This is a schematic diagram of the implementation environment of an embodiment of the present invention. The implementation environment does not limit the hardware structure of the present application, but provides a scenario example that can execute the technical solution of the present embodiment, which does not limit the specific hardware of the present application.

[0051] The implementation environment of this embodiment includes an electronic device 10 and a sound pickup device 11. The electronic device 10 can be a common computer, server or other device that can be used to run a deep learning model. The sound pickup device 11 can be a device such as a microphone that can obtain sound information. The sound pickup device 11 is connected to the electronic device 10 through a network, so that the collected ambient audio is sent to the electronic device 10 for recognition. The sound pickup device 11 can determine the specific location of the city according to the monitoring requirements of the ambient sound, and this embodiment does not limit this.

[0052] The following is based on Figure 1 The implementation environment shown is used to further illustrate the control method of the embodiment of the present invention.

[0053] Reference Figure 2 , Figure 2 A flowchart of an environmental sound recognition method based on a deep learning composite model provided in an embodiment of the present invention includes but is not limited to the following steps:

[0054] S21, obtaining ambient audio, extracting features of the ambient audio to obtain ambient audio features, and inputting the ambient audio features into a pre-trained composite model, wherein the composite model includes a plurality of trained general models, the general model is a DCNN network model, each general model has a different network structure, the general model is preset with a plurality of category labels, and the general model is preset with a model weight;

[0055] S22, traversing the composite model, classifying and identifying the environmental audio features respectively through each general model, and obtaining the classification and identification results of each general model, wherein the classification and identification results include the general identification confidence corresponding to each category label;

[0056] S23, determining a target weighted confidence of each category label based on the model weight and the general recognition confidence of the general model, and determining the target weighted confidence as the classification output result of the general model;

[0057] S24, determining a composite recognition result of the composite model according to all classification output results, wherein the composite recognition result includes a composite confidence of each category label, and the composite confidence is a mean value of all target weighted confidences corresponding to the category labels.

[0058] It should be noted that the ambient audio may be audio collected from a city, and the ambient audio may include natural sounds or non-natural sounds. Natural sounds may be animal calls or meteorological sounds, etc., and non-natural sounds may be sounds of construction work, industrial production, traffic, etc. in the city. This embodiment does not limit the specific types of sounds included in the ambient audio.

[0059] It is worth noting that the general model of this embodiment adopts a DCNN network model, so it is necessary to set a category label for each sound so that the general model can identify the specific type of environmental sound after training. For example, a category label is set for car horns, a category label is set for thunder, and a category label is set for construction noise, etc. The general model can set any number of category labels, and this embodiment does not impose any restrictions on this.

[0060] It should be noted that the composite model of this embodiment may include multiple general models, and the category labels of each general model may be the same. For the same section of ambient audio, the general recognition confidence corresponding to each category label can be obtained through multiple general models. Since the network structures of the general models are different, they can have different recognition effects on the ambient audio features. The composite confidence is obtained by weighting the model weights, which effectively reduces the risk of inaccurate recognition caused by overfitting when a model is used for a long time.

[0061] It is worth noting that the category labels of each general model may also be different. For example, some general models are configured with category labels for natural sounds, and other general models are configured with category labels for non-natural sounds. When identifying ambient audio, if the natural sounds in the ambient audio are louder, the general model configured with category labels for natural sounds has a higher weighted confidence in the target output, and the general model configured with category labels for non-natural sounds has a lower weighted confidence in the target output. The composite confidence of the category labels for natural sounds obtained after calculating the mean is higher, which can ensure the accuracy of composite model recognition.

[0062] It is worth noting that the configuration of the general model based on the DCNN network structure is a technology well known to those skilled in the art, and this embodiment does not limit the specific network structure, configuration, and training method of the model.

[0063] It should be noted that this embodiment uses a traversal method to input the environmental audio features into each general model. After obtaining the general recognition confidence of a general model, the product of the model weight and the general recognition confidence can be used as the target weighted confidence, which is saved in the result list and then traversed to the next general model. After obtaining the target weighted confidence output by all general models, all target weighted confidences of the same category label are obtained in the result list, and the composite confidence of the category label is obtained after calculating the average. The above steps are repeated for each category label, and all the composite confidences obtained are used as the output results of the composite model. In the case of composite confidence, those skilled in the art are familiar with how to complete environmental sound recognition. For example, the category label with the highest composite confidence is the sound recognition result, which will not be elaborated here.

[0064] It should be noted that the model weights can be set according to actual needs. The model weights are used to indicate the weights of the category labels in the general model. Weights can also be configured for each category label separately to achieve weighted fusion of multiple models.

[0065] Through the technical solution of this embodiment, the same section of environmental audio can be subjected to multiple heterogeneous recognitions through a composite model, and the target weighted confidence of each category label can be obtained by combining the model weights. The composite confidence of each category label is then calculated through statistical methods such as the mean, and the composite confidence obtained by weighted fusion is used as the recognition result of the environmental audio. This can effectively avoid overfitting caused by long-term operation of the same model, and effectively improve the generalization ability of the composite model.

[0066] In addition, in one embodiment, step S21 also includes but is not limited to the following steps:

[0067] S31, preprocessing the ambient audio, the preprocessing including eliminating DC components and amplitude normalization;

[0068] S32, extracting ambient audio features from the preprocessed ambient audio based on Log-Mel, PCEN or MFCCs.

[0069] It should be noted that since the ambient audio is collected directly from the city and has many sound sources, this embodiment first performs preprocessing after obtaining the ambient audio, and effectively improves the recognition accuracy and reduces the computational complexity by eliminating the DC component and normalizing the amplitude.

[0070] It should be noted that this embodiment can use one of the schemes of Log-Mel, PCEN or MFCCs for feature extraction, or can use a mixed method to extract features. For example, the ambient audio is divided into three parts, and each part uses one of the schemes to extract audio features, and then they are fused to obtain the ambient audio features. For another example, the ambient audio is simultaneously subjected to feature extraction through Log-Mel, PCEN or MFCCs, and after obtaining the three audio features, the features are fused to obtain the ambient audio features finally used, so that the ambient audio features can be represented at more levels, thereby improving the recognition accuracy of the general model.

[0071] In addition, in one embodiment, before executing step S21, the following steps are also included but not limited to:

[0072] S41, obtaining a training data set, where the training data set includes a plurality of sample audio features that are pre-labeled with category labels;

[0073] S42, dividing the training data set into a plurality of training sample sets and a plurality of test sample sets, wherein the sample audio features of each training sample set are different from each other, the sample audio features of each test sample set are different from each other, and the number of the training sample sets and the test sample sets is the same as the number of the general model;

[0074] S43, input each group of training sample sets and test sample sets into a common model, perform data enhancement on the sample audio features of the training sample sets and complete the model training.

[0075] It should be noted that the general model of this embodiment adopts a different network structure. On this basis, this embodiment further trains different general models through different training samples, so that each general model has different recognition capabilities after training. Therefore, after inputting the same piece of ambient audio, it can be recognized from different dimensions, so that the weighted composite confidence can accurately reflect the specific category of the ambient audio.

[0076] It should be noted that by training the general model with different training samples, different general models can have different recognition capabilities for the same audio feature, thereby realizing heterogeneous recognition of composite models. For example, one general model has a strong recognition capability for natural sounds, and another general model has a strong recognition capability for industrial noise. This embodiment does not limit the way of dividing the training samples, and can ensure that different general models have different recognition effects.

[0077] In addition, in one embodiment, data enhancement includes at least one of the following:

[0078] Based on a preset mixing coefficient, multiple sample audio features are randomly mixed in pairs;

[0079] Smoothing the category labels of the sample audio features;

[0080] Add Gaussian white noise with random signal-to-noise ratio to the sample audio features;

[0081] Randomly mask the time scale of the sample audio features;

[0082] Randomly mask the frequency scale of the sample audio features.

[0083] It should be noted that the above-mentioned data enhancement methods can adopt one or more methods to enhance the sample audio features during the training process, so that the significant features in the sample audio features can be enhanced, so that the trained general model can better recognize audio features.

[0084] In addition, in one embodiment, at least one general model is also associated with a special case model set, the special case model set includes at least one special case model, the special case model is a DCNN network model, each special case model has a different network structure, the special case model is preset with multiple category labels, the category label of the special case model is a true subset of the category label of the general model to which it belongs, and the category label of the special case model is preset with a label weight. Step S23 also includes but is not limited to the following steps:

[0085] S51, determining the product of the model weight and the universal recognition confidence as the universal weighted confidence;

[0086] S52, inputting the environmental audio features into the special case model for classification and recognition, and obtaining a special case recognition result output by the feature model, wherein the special case recognition result includes the special case recognition confidence of each category label of the special case model;

[0087] S53, determining the product of the special case recognition confidence and the corresponding label weight as the feature weighted confidence;

[0088] S54, when the category label is associated with a feature weighted confidence and a universal weighted confidence, a mean of the feature weighted confidence and the universal weighted confidence is determined as the target weighted confidence;

[0089] S55: When there is no associated feature weighted confidence for the category label, the universal weighted confidence is determined as the target weighted confidence.

[0090] It should be noted that this embodiment determines the product of the model weight and the general recognition confidence as the general weighted confidence, so that the general weighted confidence can form differences. For example, some general models set larger weights for the category labels of natural sounds, thereby improving the confidence of the natural sounds they identify. The specific weights can be set according to actual needs.

[0091] It should be noted that, in addition to having multiple general models for weighted recognition, this embodiment can also set multiple special case models as child nodes of the general model to enhance the recognition of some similar labels in the general model to correct the general weighted confidence obtained by the general model recognition.

[0092] It should be noted that when the category labels of each general model are the same, a set of special case models can be set for each general model, or special case models can be configured for some general models. For example, special case models can be configured for general models with longer training time and stronger recognition capabilities, so that the target weighted confidence is more accurate; when the category labels of each general model are different, they can also be configured to some general models according to the specific recognition type of the special case model. This embodiment does not limit the correspondence between the general model and the special case model, and can be set according to actual needs.

[0093] It should be noted that, in this embodiment, the same environmental audio features of the general model are used as the input of the special case model to ensure that the recognition of the special case model is consistent with that of the general model.

[0094] It should be noted that the special case model is used to enhance the recognition of specific category labels. The special case model can be deeply trained for several similar category labels to improve the ability to distinguish. Therefore, the special case model does not need to set all category labels. Its category labels are a true subset of the general model to which it belongs. For example, the general model is set with 15 categories of environmental sounds, two of which are truck whistles and ship aerodynamic sounds. Since the two have a certain similarity in audio features, the general model has limited training depth in order to cover more category labels. The ability to distinguish between truck whistles and ship aerodynamic sounds is poor, and the general weighted confidence of the two category labels is obtained. In this scenario, a special case model that is only equipped with two category labels, truck whistle and ship pneumatic sound, can be used. Through deep training, the special case model can be enabled to distinguish between truck whistle and ship pneumatic sound. When the environmental audio features are input into the special case model, the special case recognition confidence of the two category labels, truck whistle and ship pneumatic sound, is obtained. The special case weighted confidence is determined according to the product of the category weight and the special case recognition confidence. The confidence of the general model is corrected by the special case weighted confidence to obtain the target weighted confidence of the two category labels, truck whistle and ship pneumatic sound, so that the composite model can accurately identify truck whistle and ship pneumatic sound.

[0095] It should be noted that, according to the description of the above example, when the general model has a special case model, for category labels not involved in the special case model, the general weighted confidence can be determined as the target weighted confidence. For category labels designed by the special case model, the average of the general weighted confidence and the special case weighted confidence can be used as the target weighted confidence, thereby achieving confidence correction and improving accuracy.

[0096] In addition, in one embodiment, step S52 also includes but is not limited to the following steps:

[0097] S61, sorting multiple category labels from large to small according to the universal weighted confidence;

[0098] S62, determining a plurality of high confidence labels according to the order of sorting from front to back;

[0099] S63, traversing the special case model, when the high confidence label is the category label in the special case model, inputting the environmental audio features into the special case model for classification and recognition.

[0100] It should be noted that after obtaining the universal weighted confidence of the general model, this embodiment sorts from large to small according to the universal weighted confidence, and selects the top several as high-confidence labels. When the high-confidence label is a category label in the special case model, it can be determined that some relatively similar category labels have a higher confidence, and it is necessary to distinguish and identify them through the special case model to correct the confidence and improve the accuracy of composite model recognition. The number of high-confidence labels can be set according to actual needs, such as the first 3, or the first 5, and no limitation is made here.

[0101] It should be noted that this embodiment does not limit the category labels of the special case model to all be within the range of high-confidence labels. The special case model is used to correct the confidence of some category labels in the general model. Therefore, after matching the special case model, the environmental audio features can be input for recognition.

[0102] It should be noted that each general model can be configured with multiple special case models, which can be traversed according to the high confidence labels. If no category label of a special case model matches, the next high confidence label is traversed until the last high confidence label is traversed.

[0103] In addition, in one embodiment, after executing step S24, the following steps are also included but not limited to:

[0104] S71, when a new general model is obtained, configuring the new general model to the composite model, wherein the new general model is preset with multiple category labels, and at least one of the category labels of the new general model is different from the category label of the general model of the composite model;

[0105] S72, when a new special case model is obtained, the general model associated with the new special case model is determined, and the new special case model is configured into the composite model, wherein the category label of the new special case model and the category label of the special case model in the composite model do not overlap with each other;

[0106] S73, when a new general model is obtained and the new general model is associated with a new special case model, the new general model and the new special case model are configured into a composite model, wherein at least one of the category labels of the new general model is different from the category label of the general model of the composite model, and the category label of the new special case model does not overlap with the category label of the special case model in the composite model.

[0107] It should be noted that since the composite model of this embodiment adopts a multi-confidence weighted fusion method to realize ambient sound recognition, when a new sound type is added to the ambient sound, the category label can be added by adding a general model or a special case model, without the need to retrain the existing model, thereby improving the expansion convenience of the composite model.

[0108] For example, a new factory is added to the city, so industrial noise is added to the ambient sound. If the relevant technology is used, it is necessary to add a new category label of industrial noise, and then obtain training data to retrain the model. This embodiment can add a new general model and / or special case model. The category label of the general model or special case model is industrial noise. The target weighted confidence of all category labels of the previous general model is low. The new general model has a higher target weighted confidence for the category label of industrial noise. After composite fusion, the composite confidence of the category label of industrial noise is high, and the ambient sound can be identified as industrial noise. Through the technical solution of this embodiment, when it is necessary to expand a new label category, it is only necessary to train a new model with the new label and then fuse it into the composite model, which effectively reduces the model training time and improves the expansion capability of the composite model.

[0109] In order to better illustrate the technical solution of this embodiment, the following Figure 3 To provide a specific example, refer to Figure 3 This example includes but is not limited to the following steps:

[0110] S301, load and traverse the composite model, and when the next general model exists, execute steps S302 and S311, otherwise execute step S341;

[0111] S302, obtaining ambient audio, pre-processing the ambient audio, and obtaining ambient audio features through audio feature extraction;

[0112] S311, running the general model, and inputting the environmental audio features obtained in step S302 into the general model;

[0113] S312, identifying the environmental audio features through a universal model to obtain a universal recognition confidence;

[0114] S313, determining the product of the model weight and the universal recognition confidence as the universal weighted confidence, and when the universal model is configured with a special case model, executing step S321, otherwise executing step S331;

[0115] S321, take the first m largest ones among the general weighted confidences, determine the corresponding category label as a high confidence label, and when there is a next high confidence label and there is a corresponding special case model for the high confidence label, execute step S322, otherwise execute step S331;

[0116] S322, running the special case model, and inputting the environmental audio features obtained in step S302 into the special case model;

[0117] S323, performing classification and identification through the special case model to obtain the special case identification confidence;

[0118] S324, determining the product of the tag weight and the special case identification confidence as the special case weighted confidence;

[0119] S325, correcting the universal weighted confidence of the same category label according to the special case weighted confidence;

[0120] S331, save the universal weighted confidence, and when the composite model is traversed, execute step S341, otherwise traverse to the next universal model and execute step S311;

[0121] S341, taking the average of the universal weighted confidences of the same category labels, and using the obtained target weighted confidence as the model output of the composite model.

[0122] like Figure 4 As shown, Figure 4 : is a structural diagram of an environmental sound recognition device based on a deep learning composite model provided by an embodiment of the present invention. The present invention also provides an environmental sound recognition device based on a deep learning composite model, comprising:

[0123] The processor 401 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0124] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the environmental sound recognition method based on the deep learning composite model of the embodiment of the present application;

[0125] Input / output interface 403, used to implement information input and output;

[0126] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0127] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0128] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0129] An embodiment of the present application also provides an electronic device, including the above-mentioned environmental sound recognition device based on the deep learning composite model.

[0130] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned environmental sound recognition method based on the deep learning composite model is implemented.

[0131] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0132] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0133] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for environmental sound recognition based on a deep learning composite model, characterized in that: include: Acquire ambient audio, extract features of the ambient audio to obtain ambient audio features, and input the ambient audio features into a pre-trained composite model, wherein the composite model includes multiple trained general models, the general model is a DCNN network model, each general model has a different network structure, the general model is preset with multiple category labels, and the general model is preset with model weights; Traversing the composite model, classifying and identifying the environmental audio features through each general model, and obtaining the classification and identification results of each general model, wherein the classification and identification results include the general identification confidence corresponding to each category label; Based on the model weights and the universal recognition confidence of the universal model, the target weighted confidence of each category label is determined, and the target weighted confidence is determined as the classification output result of the universal model; Determine the composite recognition result of the composite model according to all the classification output results, wherein the composite recognition result includes the composite confidence of each category label, and the composite confidence is the average of all the target weighted confidences corresponding to the category label; At least one general model is also associated with a special case model set, the special case model set includes at least one special case model, the special case model is a DCNN network model, each special case model has a different network structure, the special case model is preset with multiple category labels, the category label of the special case model is a true subset of the category label of the general model to which it belongs, the category label of the special case model is preset with a label weight, based on the model weight and the general confidence of the general model, the target weighted confidence of each category label is determined, including: The product of the model weight and the universal recognition confidence is determined as the universal weighted confidence; Inputting the environmental audio features into the special case model for classification and recognition, obtaining the special case recognition result output by the feature model, wherein the special case recognition result includes the special case recognition confidence of each category label of the special case model; The product of the special case recognition confidence and the corresponding label weight is determined as the feature weighted confidence; When the category label is associated with a feature weighted confidence and a universal weighted confidence, the mean of the feature weighted confidence and the universal weighted confidence is determined as the target weighted confidence; Alternatively, when there is no associated feature weighted confidence for the category label, the universal weighted confidence is determined as the target weighted confidence.

2. The method for environmental sound recognition based on a deep learning composite model according to claim 1, characterized in that: The feature extraction of the ambient audio obtains the ambient audio features, including: Preprocess the ambient audio, including eliminating DC components and normalizing the amplitude; Based on Log-Mel, PCEN or MFCCs, the ambient audio features are extracted from the preprocessed ambient audio.

3. The method for environmental sound recognition based on a deep learning composite model according to claim 1, characterized in that: Before obtaining the ambient audio, the method also includes: Obtain a training data set, where the training data set includes multiple sample audio features with pre-labeled category labels; The training data set is divided into a plurality of training sample sets and a plurality of test sample sets, wherein the sample audio features of each training sample set are different from each other, the sample audio features of each test sample set are different from each other, and the number of the training sample sets and the test sample sets is the same as the number of the general model; Each set of training sample sets and test sample sets is input into a common model, and the model training is completed after data enhancement of the sample audio features of the training sample sets.

4. The method for environmental sound recognition based on a deep learning composite model according to claim 3, characterized in that: Data augmentation includes at least one of the following: Based on a preset mixing coefficient, multiple sample audio features are randomly mixed in pairs; Smoothing the category labels of the sample audio features; Add Gaussian white noise with random signal-to-noise ratio to the sample audio features; Randomly mask the time scale of sample audio features; Randomly mask the frequency scale of the sample audio features.

5. The method for environmental sound recognition based on a deep learning composite model according to claim 1, characterized in that: Input the environmental audio features into the special case model for classification and recognition, including: Sort multiple category labels from large to small according to the universal weighted confidence; Determine multiple high-confidence labels according to the order of sorting from front to back; Traverse the special case model, and when the high confidence label is the category label in the special case model, input the environmental audio features into the special case model for classification and recognition.

6. The method for environmental sound recognition based on a deep learning composite model according to claim 1, characterized in that: After determining the composite recognition result of the composite model according to all the classification output results, the method further includes: When a new general model is obtained, the new general model is configured into the composite model, wherein the new general model is preset with multiple category labels, and at least one of the category labels of the new general model is different from the category label of the general model of the composite model; Alternatively, when a new special case model is obtained, the general model associated with the new special case model is determined, and the new special case model is configured into the composite model, wherein the category labels of the new special case model and the category labels of the special case model in the composite model do not overlap with each other; Alternatively, when a new general model is obtained and the new general model is associated with a new special case model, the new general model and the new special case model are configured into a composite model, wherein at least one of the category labels of the new general model is different from the category label of the general model of the composite model, and the category label of the new special case model does not overlap with the category label of the special case model in the composite model.

7. An environmental sound recognition device based on a deep learning composite model, characterized in that: comprising at least one control processor and a memory for communicatively coupling with the at least one control processor; The memory stores instructions that can be executed by at least one control processor, and the instructions are executed by at least one control processor to enable the at least one control processor to execute the environmental sound recognition method based on the deep learning composite model as claimed in any one of claims 1 to 6.

8. An electronic device, characterized in that: An environmental sound recognition device based on a deep learning composite model including claim 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to enable a computer to execute the environmental sound recognition method based on a deep learning composite model as claimed in any one of claims 1 to 6.

Images