Snore sound recognition method and device, computer device and computer readable storage medium

Abstract

The application discloses a snoring sound recognition method and device, computer equipment and a computer readable storage medium, and relates to the field of sound recognition. The method comprises the following steps: a computer equipment acquires sound data of a user during sleep, inputs the sound data into a first convolution module of a preset neural network to obtain a first feature output; meanwhile, the sound data is spectrally processed, and the spectral processing result is input into a second convolution module of the preset neural network to obtain a second feature output; then, the first feature output and the second feature output are spliced to obtain a fusion feature output; finally, a classification module of the preset neural network is controlled to perform snoring sound prediction based on the fusion feature output to obtain a snoring sound recognition result corresponding to the sound. Based on this, the sound data is subjected to feature extraction in different dimensions in the embodiment of the application, so that the feature expression corresponding to the sound data is more abundant, and then the classification module can complete high-precision snoring sound recognition according to the abundant features.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sound recognition, and in particular to a snoring sound recognition method and device, computer equipment and a computer readable storage medium. BACKGROUND

[0002] In existing snoring sound recognition, a snoring sound is usually recognized by judging whether the features of the real snoring sound match the features of the sound signal emitted by the user during sleep, such as duration, interval time, breathing frequency, and interval period.

[0003] However, the selection of snoring sound features is highly dependent on experience, and the snoring sound features of users in different age groups or different sleep environments differ, which makes it difficult to apply a snoring sound recognition method to multiple user groups, resulting in low efficiency and accuracy of the snoring sound recognition method. SUMMARY

[0004] Therefore, the present application provides a snoring sound recognition method, device, computer equipment and computer readable storage medium to improve the low efficiency and accuracy of the snoring sound recognition method.

[0005] In a first aspect, the present application provides a snoring sound recognition method, comprising:

[0006] obtaining sound data of a user during sleep;

[0007] inputting the sound data into a first convolution module of a preset neural network to obtain a first feature output;

[0008] performing a frequency spectrum processing on the sound data, and inputting the frequency spectrum processing result into a second convolution module of the preset neural network to obtain a second feature output;

[0009] concatenating the first feature output and the second feature output to obtain a fusion feature output;

[0010] using a classification module of the preset neural network to predict the fusion feature output to obtain a snoring sound recognition result corresponding to the sound.

[0011] Optionally, in one feasible manner provided by the present application, the first convolution module comprises a plurality of first convolution sub-modules and a global average pooling layer connected in sequence, and the first convolution sub-module comprises a one-dimensional convolution layer, a first activation function layer, a batch normalization layer and a first maximum pooling layer connected in sequence.

[0012] The sound data is inputted into the first convolution module of the preset neural network to obtain the first feature output, comprising:

[0013] Input the sound data into the plurality of first convolution sub-modules to perform feature extraction, linear activation, normalization and dimension reduction processing step by step, to obtain data feature representation;

[0014] Downsample the data feature representation by using the global average pooling layer to obtain first feature output.

[0015] Optionally, in an available manner provided by the embodiment of the present application, the spectrum processing is performed on the sound data, and the spectrum processing result is input into the second convolution module of the preset neural network to obtain second feature output, including:

[0016] Performing processing on the sound data based on mel frequency cepstral coefficient to obtain spectrum feature output;

[0017] Input the spectrum feature output into the second convolution module of the preset neural network to obtain second feature output.

[0018] Further, in an available manner provided by the embodiment of the present application, the second convolution module includes a plurality of second convolution sub-modules connected in sequence, and the second convolution sub-module includes a two-dimensional convolution layer, a second activation function layer and a second maximum pooling layer connected in sequence;

[0019] The input of the spectrum feature output into the second convolution module of the preset neural network to obtain second feature output includes:

[0020] Input the spectrum feature output into the plurality of second convolution sub-modules connected in sequence to perform step-by-step convolution and pooling to obtain second feature output.

[0021] Optionally, in an available manner provided by the embodiment of the present application, the splicing of the first feature output and the second feature output to obtain fusion feature output includes:

[0022] Performing flattening processing on the second feature output;

[0023] Splicing the first feature output and the second feature output after flattening processing to obtain fusion feature output.

[0024] Optionally, in an available manner provided by the embodiment of the present application, the splicing of the first feature output and the second feature output to obtain fusion feature output includes:

[0025] Adjusting the dimension size of the first feature output to make the dimension size of the adjusted first feature output same as the second feature output;

[0026] Splicing the second feature output and the adjusted first feature output to obtain fusion feature output.

[0027] Optionally, in an implementation provided by the embodiment of the present application, the preset neural network is trained by multiple noise data and sound data of multiple users during sleep.

[0028] Optionally, in an implementation provided by the embodiment of the present application, the method further comprises:

[0029] When the multiple pieces of sound data of the user during sleep are acquired, the preset neural network is updated by using the multiple pieces of sound data.

[0030] In a second aspect, an embodiment of the present application provides a snoring sound recognition device, comprising:

[0031] an acquisition module, configured to acquire sound data of a user during sleep;

[0032] a first convolution processing module, configured to input the sound data into a first convolution module of a preset neural network to obtain a first feature output;

[0033] a second convolution processing module, configured to perform spectrum processing on the sound data and input a spectrum processing result into a second convolution module of the preset neural network to obtain a second feature output;

[0034] a splicing module, configured to splice the first feature output and the second feature output to obtain a fusion feature output;

[0035] a recognition module, configured to use a classification module of the preset neural network to predict the fusion feature output to obtain a snoring sound recognition result corresponding to the sound.

[0036] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program performs the snoring sound recognition method disclosed in any one of the first aspect when running on the processor.

[0037] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program performs the snoring sound recognition method disclosed in any one of the first aspect when running on the processor.

[0038] In the snoring sound recognition method provided by the embodiment of the present application, after the computer device obtains the sound data of the user during sleep, the sound data is input to the first convolution module of the preset neural network to extract the feature information of the sound in the semantic and data distribution, i.e., the first feature output; at the same time, the sound data is spectrally processed, and the spectral processing result is input to the second convolution module of the preset neural network to extract the spectral feature of the sound, i.e., the second feature output is obtained; then, the first feature output and the second feature output are spliced to obtain the fusion feature output; finally, the classification module of the preset neural network is controlled to perform snoring sound prediction according to the fusion feature output to determine whether the sound data contains snoring sound, and obtain the snoring sound recognition result corresponding to the sound data.

[0039] Based on this, the embodiment of the present application realizes feature extraction of sound data in different dimensions, so that the feature expression corresponding to the sound data is more abundant, and then the classification module can perform high-precision snoring sound recognition according to the abundant feature expression. Moreover, because the neural network model will adaptively determine the important information in the sound data according to the actual result and the predicted result in the training process, the limitation of manually selecting snoring sound features can be effectively avoided, so that the preset neural network has strong anti-interference performance, and thus can provide effective snoring sound recognition for snoring sound recognition of different user groups. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope of protection of the present application. In each drawing, similar components are marked with similar reference numerals.

[0041] Figure 1 A flowchart of a first snoring sound recognition method provided by the embodiment of the present application is shown;

[0042] Figure 2 A flowchart of a second snoring sound recognition method provided by the embodiment of the present application is shown;

[0043] Figure 3 A running schematic diagram of the first convolution module provided by the embodiment of the present application is shown;

[0044] Figure 4 A first dimension adjustment schematic diagram provided by the embodiment of the present application is shown;

[0045] Figure 5 A second dimension adjustment schematic diagram provided by the embodiment of the present application is shown;

[0046] Figure 6 A running schematic diagram of the second convolution module provided by the embodiment of the present application is shown;

[0047] Figure 7 A structure diagram of a snoring sound recognition device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application.

[0049] The components of the embodiments of the present application generally described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0050] Hereinafter, the terms "include", "have", and their conjugates used in various embodiments of the present application are only intended to denote a certain characteristic, number, step, operation, element, component, or combination of the foregoing, and should not be construed as excluding the presence or addition of one or more other characteristics, numbers, steps, operations, elements, components, or combinations thereof.

[0051] In addition, the terms "first", "second", "third", and the like are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0052] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present application belong. The terms (such as those defined in a generally used dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized or overly formal meaning, unless clearly defined in various embodiments of the present application.

[0053] Embodiment 1

[0054] Referring to Figure 1 , a flowchart of a first snoring sound recognition method provided by an embodiment of the present application is shown, and the snoring sound recognition method provided by the embodiment of the present application includes:

[0055] S110, acquiring sound data of a user during sleep.

[0056] It can be understood that the manner of acquiring the sound data of the user during sleep can be set according to actual conditions, such as in a feasible manner, the embodiment of the present application acquires the sound data of the user during sleep through a microphone arranged on one side of a pillow of the user.

[0057] In a preferred manner, the embodiment of the present application is provided with a sound collector, such as a microphone, in the pillow of the user in advance to realize efficient collection of sound data and ensure that snoring sound emitted by the user can be collected.

[0058] In addition, it can be understood that the length of the sound data can also be set according to actual conditions, such as in a feasible manner provided by the embodiment of the present application, the length of the sound data is 1 second to ensure effective collection of snoring sound.

[0059] S120, inputting the sound data into a first convolution module of a preset neural network to obtain a first feature output.

[0060] That is, the computer device in the embodiment of the present application extracts the first feature output of the sound data, that is, the feature information on data distribution and semantics, by using the first convolution module of the preset neural network, so that the classification module of the preset neural network classifies snoring sound according to the feature information on data distribution and semantics of the sound data.

[0061] It can be understood that the specific structure of the first convolution module can be set according to actual conditions, such as in an optional manner, the embodiment of the present application realizes the first convolution module based on a preset ResNet (Residual Network) model.

[0062] In another implementable manner of the first convolution module provided by the embodiment of the present application, please refer to Figure 2 , which shows a flowchart of a second snoring sound recognition method provided by the embodiment of the present application, that is, in this feasible manner, the first convolution module includes a plurality of first convolution sub-modules and a global average pooling layer connected in sequence, the first convolution sub-module includes a one-dimensional convolution layer, a first activation function layer, a batch normalization layer and a first maximum pooling layer connected in sequence;

[0063] Further, the S120 includes:

[0064] S121, inputting the sound data into the plurality of first convolution sub-modules to gradually perform feature extraction, linear activation, normalization and dimension reduction processing to obtain data feature representation;

[0065] S122, using the global average pooling layer to down-sample the data feature representation to obtain the first feature output.

[0066] To better illustrate the implementation of the first convolution module in the embodiments of the present application, please refer to Figure 3 , which shows a running schematic diagram of the first convolution module provided by the embodiments of the present application. In the diagram, conv_1d corresponds to a one-dimensional convolution layer, BN corresponds to a batch normalization (Batch Normalization) layer, maxpool_1d corresponds to a first max pooling layer, and gobal avg_pooling_2d corresponds to a global average pooling layer. Among them, each 1d above represents one-dimensional operation, and 2d represents two-dimensional operation.

[0067] It should be noted that, Figure 3 The convolution kernel size, the max pooling window size and the global average pooling window size in the first convolution module shown are all examples, and can be set according to actual conditions.

[0068] Based on this, the embodiments of the present application are based on the batch normalization processing and the pooling processing in the first convolution module as shown in Figure 3 , so that the semantic features and the data distribution features in the sound data can be efficiently collected, and the problem of gradient disappearance caused by the deep network can be avoided.

[0069] In addition, it can be understood that the convolution kernel size and the pooling window size in the first convolution module can be set according to actual conditions. For example, in the feasible manner provided by the embodiments of the present application, the convolution kernel size in the first convolution module is 3x3, and the max / global average pooling window size in the first convolution module is 3x2 or 4x4.

[0070] S130, performing a spectral processing on the sound data, and inputting the spectral processing result into a second convolution module of the preset neural network to obtain a second feature output.

[0071] That is, the computer device in the embodiments of the present application extracts not only the feature information of the sound data in the data distribution and the semantics, but also the spectral feature of the sound data, i.e., the second feature output.

[0072] It can be understood that the spectrum corresponding to the sound can be obtained after the mechanical vibration corresponding to the sound is decomposed, i.e., the sound spectrum. The spectrum can intuitively display the amplitude and frequency of the mechanical vibration corresponding to the sound, and thus can effectively reflect the internal structure of the sound signal.

[0073] Further, the embodiment of the present application enables the classification module to perform snoring sound recognition based on richer feature information, and therefore, the sound data is subjected to spectral processing to extract the spectral features corresponding to the sound data, i.e., the second feature output. In this way, the computer device in the embodiment of the present application extracts the feature information of the sound data in terms of data distribution and semantics, and also extracts the spectral feature information corresponding to the sound data, so that the classification module can complete snoring sound recognition according to the rich feature information, thereby ensuring the accuracy of snoring sound recognition.

[0074] In addition, it can be understood that the spectral processing manner can be set according to actual conditions. For example, in a feasible manner, the embodiment of the present application implements the spectral processing of the sound data through a preset spectrogram to quickly complete the spectral processing, thereby improving the inference speed of the preset neural network.

[0075] In a preferred spectral processing manner provided by the embodiment of the present application, the spectral processing is completed based on Mel-frequency cepstral coefficients (MFCC), i.e., the S130 includes:

[0076] The sound data is processed based on the Mel-frequency cepstral coefficients to obtain a spectral feature output;

[0077] The spectral feature output is input into a second convolution module of the preset neural network to obtain a second feature output.

[0078] It can be understood that the Mel-frequency cepstral coefficients are used to create a mel-cepstrum. Unlike the general cepstrum, the frequency bands of the mel-cepstrum are uniformly distributed on the mel scale, which is very close to the human auditory system.

[0079] Therefore, the embodiment of the present application performs spectral processing of the sound data based on the Mel-frequency cepstral coefficients, so that the sound features of the human body are more obvious, and the computer device can extract effective spectral features.

[0080] S140, the first feature output and the second feature output are spliced to obtain a fusion feature output.

[0081] That is, the computer device in the embodiment of the present application splices the first feature output and the second feature output, so that the classification module can perform snoring sound recognition based on the rich feature information of the sound data in the spliced result, i.e., the fusion feature output.

[0082] It can be understood that the specific splicing manner of the first feature output and the second feature output can be set according to actual conditions.

[0083] Optionally, in a feasible manner provided by the embodiment of the present application, the S130 includes:

[0084] performing a flattening process on the second feature output;

[0085] splicing the first feature output and the second feature output after the flattening process to obtain a fusion feature output.

[0086] That is, the computer device will perform a flattening process on the second feature output to make the dimension of the second feature output the same as that of the first feature output; then, the second feature output after the flattening process is spliced after the first feature output to obtain a fusion feature output.

[0087] It is not difficult to understand that this splicing method will result in the loss of part of the spectral feature, and therefore, the fusion feature output obtained based on this splicing method pays more attention to the information in the first feature output, that is, pays more attention to the feature information of the sound data in the data distribution and the semantics, and therefore, if the first feature output can better reflect the snoring feature, this splicing method can achieve higher snoring recognition accuracy.

[0088] In another feasible manner provided by the embodiment of the present application, the fusion feature output pays more attention to the second feature output, that is, the S140 comprises:

[0089] adjusting the dimension size of the first feature output to make the dimension size of the adjusted first feature output the same as that of the second feature output;

[0090] splicing the second feature output and the adjusted first feature output to obtain a fusion feature output.

[0091] That is, in this feasible manner, the embodiment of the present application adjusts the dimension of the first feature output to make the dimension of the first feature output the same as that of the second feature output, and then splices the second feature output and the first feature output after the dimension adjustment to obtain a fusion feature output.

[0092] It can be understood that this feasible manner pays more attention to the information in the second feature output, and therefore, can achieve a snoring recognition result with higher accuracy when the spectral feature has higher credibility.

[0093] It can also be understood that the manner of adjusting the dimension of the first feature output can be set according to actual conditions, for example, in a feasible manner of dimension adjustment provided by the embodiment of the present application, please refer to Figure 4 , which shows a first dimension adjustment schematic diagram provided by the embodiment of the present application. In the figure, 141 is the first feature output, 142 is the second feature output, and 143 is the first dimension adjustment manner corresponding to the adjusted first feature output.

[0094] from Figure 4It can be seen that in this way, the embodiment of the present invention will adjust the width of the first feature output 141 so that the adjusted first feature output 143 is consistent with the width of the second feature output 142, so that the first feature output 141 and the second feature output 142 are consistent in the width dimension.

[0095] In another feasible method of dimensional adjustment provided by the embodiment of the present invention, please refer to Figure 5 , shows a schematic diagram of the second dimension adjustment provided by an embodiment of the present invention. In the figure, 141 is the first feature output, 142 is the second feature output, and 144 is the adjusted first feature output corresponding to the second dimension adjustment method.

[0096] Depend on Figure 5 It can be seen that under this feasible method, the embodiment of the present invention not only adjusts the width of the first feature output 141, but also adjusts the height of the first feature output 141, so that the adjusted first feature output 144 and the second feature output 142 are consistent in both width and height dimensions.

[0097] S150: Utilize the classification module of the preset neural network to predict the fusion feature output to obtain a snoring recognition result corresponding to the sound.

[0098] It can be understood that the classification module is used to determine whether the user snores during sleep, that is, whether the sound data contains snoring, based on the fusion feature output obtained above.

[0099] It can also be understood that the specific structure of the classification module can be set according to actual conditions. For example, in one feasible manner provided in an embodiment of the present invention, the classification module includes a plurality of classification units connected in sequence, and the classification unit includes a first fully connected layer, a loss layer, and a second fully connected layer connected in sequence;

[0100] Furthermore, the S150 includes:

[0101] Calculating the classification output corresponding to the fusion feature output using the plurality of classification units connected in sequence;

[0102] The classification output is input into a preset classification function to obtain a snoring recognition result corresponding to the sound.

[0103] It's understood that the dropout layer is used to randomly discard some neurons during the neural network inference process, thereby reducing the computer resources required for neural network inference and increasing the inference speed of the neural network. Furthermore, due to the configuration of the dropout layer, the preset neural network can sequentially train the discarded and non-discarded neurons during each batch size of sample training, effectively preventing overfitting of the preset neural network during training.

[0104] It can also be understood that the classification function can be set according to actual conditions, such as a feasible manner provided in the embodiment of the present application, the classification function is a Sigmoid function.

[0105] In the snoring sound recognition method provided by the embodiment of the present application, after the computer device obtains the sound data of the user during sleep, the sound data is input into the first convolution module of the preset neural network to extract the feature information of the sound in the semantic and data distribution, that is, the first feature output; at the same time, the sound data is spectrally processed, and the spectral processing result is input into the second convolution module of the preset neural network to extract the spectral feature of the sound, that is, the second feature output is obtained; then, the first feature output and the second feature output are spliced to obtain the fusion feature output; finally, the classification module of the preset neural network is controlled to perform snoring sound prediction according to the fusion feature output to judge whether the sound data contains snoring sound, and obtain the snoring sound recognition result corresponding to the sound data.

[0106] Based on this, the embodiment of the present application realizes feature extraction of sound data in different dimensions, so that the feature expression corresponding to the sound data is more abundant, and then the classification module can perform high-precision snoring sound recognition according to the abundant feature expression. Moreover, because the neural network model will adaptively determine the important information in the sound data according to the actual result and the predicted result in the training process, the limitation of manually selecting snoring sound features can be effectively avoided, so that the preset neural network has strong anti-interference performance, and thus can provide effective snoring sound recognition for snoring sound recognition of different user groups.

[0107] Optionally, in a feasible manner provided by the embodiment of the present application, the second convolution module includes a plurality of second convolution sub-modules connected in sequence, and the second convolution sub-module includes a two-dimensional convolution layer, a second activation function layer and a second maximum pooling layer connected in sequence.

[0108] Further, the S120 includes:

[0109] The spectral feature output is input into the plurality of second convolution sub-modules connected in sequence to perform step-by-step convolution and pooling to obtain the second feature output.

[0110] To better illustrate this feasible manner provided by the embodiment of the present application, please refer to Figure 6 , which shows a running schematic diagram of the second convolution module provided by the embodiment of the present application, in which, conv_2d represents a two-dimensional convolution layer, and maxpool_2d represents a second maximum pooling layer. Among them, 2d represents two dimensions. It should be noted that the second activation function layer is not shown in Figure 6The feature map is shown in the figure, but in the actual production environment, the feature map needs to pass through the conv_2d for convolution operation, and then pass through the second activation function layer for function function, and then pass through the maxpool_2d for maximum pooling.

[0111] It can be understood that the convolution kernel size and the pooling window size in the second convolution module can be set according to actual conditions. For example, in one feasible manner provided in the embodiment of the present application, the convolution kernel size in the second convolution module is 3x3, and the maximum pooling window size in the second convolution module is 3x2 or 4x4.

[0112] Optionally, in one feasible manner provided in the embodiment of the present application, the preset neural network is obtained by training a plurality of noise data and a plurality of sound data of users during sleep.

[0113] It can be understood that if a user sleeps in a house near a noisy street, the sound data collected will inevitably contain noise in the street, and the noise will have a negative impact on snoring recognition.

[0114] Therefore, in order to improve the ability of the preset neural network to recognize snoring sound from sound data in a noisy environment, a plurality of noise data is used as a snoring negative example to assist the training of the preset neural network, thereby improving the anti-interference ability of the preset neural network.

[0115] Meanwhile, it can be understood that although the sound data output by the first convolution module and the second convolution module in the embodiment of the present application has different dimensional feature information to improve the anti-interference ability of the preset neural network, because the snoring sounds of different users may be different, in order to further improve the generalization ability of the preset neural network, a plurality of sound data of users during sleep is used to train the preset neural network.

[0116] Optionally, in one feasible manner provided in the embodiment of the present application, the method further comprises:

[0117] When the plurality of sound data of the user during sleep is obtained, the plurality of sound data is used to update the preset neural network.

[0118] That is, in order to make the snoring recognition ability of the preset neural network more targeted, after each sound data of the user during sleep is collected, the sound data is stored, and after a sufficient amount of sound data is stored, the stored sound data is used to upgrade the preset neural network, so that the preset neural network can fully learn the sound characteristics of the user and clearly identify the noise of the environment in which the user is located, thereby accurately recognizing the snoring sound of the user.

[0119] Embodiment 2

[0120] Corresponding to the snoring sound recognition method provided by the embodiment 1 of the present application, the embodiment 2 of the present application further provides a snoring sound recognition device, referring to Figure 7 , a structure schematic diagram of the snoring sound recognition device provided by the embodiment of the present application is shown, the snoring sound recognition device 200 provided by the embodiment of the present application comprises:

[0121] The acquisition module 210 is configured to acquire sound data of a user during sleep.

[0122] The first convolution processing module 220 is configured to input the sound data into a first convolution module of a preset neural network to obtain a first feature output.

[0123] The second convolution processing module 230 is configured to perform spectrum processing on the sound data and input a spectrum processing result into a second convolution module of the preset neural network to obtain a second feature output.

[0124] The splicing module 240 is configured to splice the first feature output and the second feature output to obtain a fusion feature output.

[0125] The recognition module 250 is configured to use a classification module of the preset neural network to predict the fusion feature output to obtain a snoring sound recognition result corresponding to the sound.

[0126] Optionally, in a feasible manner provided by the embodiment of the present application, the first convolution module comprises a plurality of first convolution sub-modules and a global average pooling layer connected in sequence, and the first convolution sub-module comprises a one-dimensional convolution layer, a first activation function layer, a batch normalization layer and a first maximum pooling layer connected in sequence.

[0127] The first convolution processing module comprises:

[0128] The data feature representation acquisition sub-module is configured to input the sound data into the plurality of first convolution sub-modules to gradually perform feature extraction, linear activation, normalization and dimension reduction processing to obtain a data feature representation.

[0129] The down-sampling processing sub-module is configured to use the global average pooling layer to down-sample the data feature representation to obtain the first feature output.

[0130] Optionally, in a feasible manner provided by the embodiment of the present application, the second convolution processing module comprises:

[0131] The spectrum processing sub-module is configured to process the sound data based on a mel-frequency cepstral coefficient to obtain a spectrum feature output.

[0132] The feature extraction submodule is configured to input the spectral feature output into a second convolution module of the preset neural network to obtain a second feature output.

[0133] Optionally, in an implementation provided by the embodiment of the present application, the second convolution module comprises a plurality of second convolution submodules connected in sequence, and each second convolution submodule comprises a two-dimensional convolution layer, a second activation function layer and a second maximum pooling layer connected in sequence.

[0134] The feature extraction submodule comprises:

[0135] The spectral data feature acquisition unit is configured to input the spectral feature output into the plurality of second convolution submodules connected in sequence for step-by-step convolution and pooling to obtain a second feature output.

[0136] Optionally, in an implementation provided by the embodiment of the present application, the splicing module comprises:

[0137] The flattening processing submodule is configured to perform flattening processing on the second feature output.

[0138] The output splicing submodule is configured to splice the first feature output and the second feature output after flattening processing to obtain a fusion feature output.

[0139] Optionally, in an implementation provided by the embodiment of the present application, the splicing module comprises:

[0140] The adjusting submodule is configured to adjust the dimension size of the first feature output so that the dimension size of the adjusted first feature output is the same as that of the second feature output.

[0141] The feature splicing submodule is configured to splice the second feature output and the adjusted first feature output to obtain a fusion feature output.

[0142] Optionally, in an implementation provided by the embodiment of the present application, the preset neural network is trained by a plurality of noise data and a plurality of sound data of users during sleep.

[0143] Optionally, in an implementation provided by the embodiment of the present application, the device further comprises:

[0144] The updating module is configured to update the preset neural network by using the plurality of sound data of the user during sleep when the plurality of sound data of the user during sleep is acquired.

[0145] The snoring sound recognition device provided by the embodiment of the present application can implement each process of the snoring sound recognition method corresponding to the embodiment 1 and achieve the same technical effects. To avoid repetition, details are not described herein.

[0146] The embodiment of the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the computer program performs the snoring sound recognition method corresponding to the embodiment 1 when running on the processor.

[0147] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program performs the snoring sound recognition method corresponding to the embodiment 1 when running on the processor.

[0148] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are only schematic, for example, the flow charts and structural diagrams in the drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flow charts or structural diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in the alternative implementation manners, the functions annotated in the blocks can also occur in the order different from that annotated in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can also be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the structural diagram and / or flow chart, and the combination of blocks in the structural diagram and / or flow chart, can be implemented by a dedicated hardware-based system for implementing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0149] In addition, each functional module or unit in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0150] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A snoring sound recognition method characterized by, The method comprises: acquiring sound data of a user during sleep; inputting the sound data into a first convolution module of a preset neural network to obtain a first feature output, the first feature output comprising semantic features and data distribution features in the sound data; performing spectral processing on the sound data and inputting a spectral processing result into a second convolution module of the preset neural network to obtain a second feature output, the second feature output being spectral features showing amplitudes and frequencies of mechanical vibrations in the sound data; splicing the first feature output and the second feature output to obtain a fusion feature output; using a classification module of the preset neural network to predict the fusion feature output to obtain a snoring sound recognition result corresponding to the sound.

2. The snoring sound recognition method according to claim 1, characterized by, The first convolution module comprises a plurality of first convolution sub-modules and a global average pooling layer connected in sequence, and the first convolution sub-module comprises a one-dimensional convolution layer, a first activation function layer, a batch normalization layer and a first maximum pooling layer connected in sequence. The inputting of the sound data into the first convolution module of the preset neural network to obtain the first feature output comprises: inputting the sound data into the plurality of first convolution sub-modules to gradually perform feature extraction, linear activation, normalization and dimension reduction processing to obtain a data feature representation; using the global average pooling layer to down-sample the data feature representation to obtain the first feature output.

3. The snoring sound recognition method of claim 1, wherein, The spectral processing of the sound data and the inputting of a spectral processing result into the second convolution module of the preset neural network to obtain the second feature output comprise: processing the sound data based on mel-frequency cepstral coefficients to obtain a spectral feature output; inputting the spectral feature output into the second convolution module of the preset neural network to obtain the second feature output.

4. The snoring sound recognition method according to claim 3, characterized by, The second convolution module comprises a plurality of second convolution sub-modules connected in sequence, and the second convolution sub-module comprises a two-dimensional convolution layer, a second activation function layer and a second maximum pooling layer connected in sequence. The inputting of the spectral feature output into the second convolution module of the preset neural network to obtain the second feature output comprises: inputting the spectral feature output into the plurality of second convolution sub-modules connected in sequence to gradually perform convolution and pooling to obtain the second feature output.

5. The snoring sound recognition method of claim 1, wherein, The splicing of the first feature output and the second feature output to obtain the fusion feature output comprises: performing flattening processing on the second feature output; splicing the first feature output and the flattened second feature output to obtain the fusion feature output.

6. The snoring sound recognition method of claim 1, wherein, The splicing of the first feature output and the second feature output to obtain the fusion feature output comprises: adjusting a dimension size of the first feature output so that the dimension size of the adjusted first feature output is the same as that of the second feature output; splicing the second feature output and the adjusted first feature output to obtain the fusion feature output.

7. The snoring sound recognition method of claim 1, wherein, The preset neural network is trained by a plurality of noise data and sound data of a plurality of users during sleep.

8. The snoring sound recognition method of claim 1, wherein, The method further comprises: When a plurality of pieces of sound data of the user during sleep are acquired, the plurality of pieces of sound data are used to update the preset neural network.

9. A snoring sound recognition apparatus characterized by comprising: The method comprises the following steps: acquiring sound data of a user during sleep; a first convolution processing module is configured to input the sound data into a first convolution module of a preset neural network to obtain a first feature output, wherein the first feature output comprises semantic features and data distribution features in the sound data; a second convolution processing module is configured to perform spectral processing on the sound data and input a spectral processing result into a second convolution module of the preset neural network to obtain a second feature output, wherein the second feature output is a spectral feature that displays amplitudes and frequencies of mechanical vibrations in the sound data; a splicing module is configured to splice the first feature output and the second feature output to obtain a fusion feature output; a recognition module is configured to use a classification module of the preset neural network to predict the fusion feature output to obtain a snoring sound recognition result corresponding to the sound.

10. A computer device, comprising: The computer readable storage medium stores a computer program, and the computer program performs the snoring sound recognition method of any one of claims 1-8 when running on the processor.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program performs the snoring sound recognition method of any one of claims 1-8 when running on the processor.

Images