A sleep sound recognition method and device, electronic equipment and storage medium

Abstract

The application provides a sleep sound identification method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining sound data, and determining valid data from the sound data; performing sleep sound identification on the valid data to obtain a sleep sound identification result. In the implementation process of the above scheme, the valid data is determined from the sound data, and the sleep sound identification is performed on the valid data, which improves the identification of all sleep sounds, effectively reduces the calculation power consumption and calculation resource consumption of sleep sound identification.

Description

Technical Field

[0001] This application relates to the technical fields of sound recognition and neural networks, and more specifically, to a sleep sound recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Sleep sound recognition refers to classifying and identifying sounds collected during sleep to obtain sound categories. These sound categories include, but are not limited to, snoring, sleep talking, breathing, turning over, teeth grinding, coughing, and environmental sounds such as cat, dog, bird calls, baby crying, car horns, and rain sounds.

[0003] Currently, due to the large computational demands of sleep sound recognition, most methods involve collecting and transmitting data to a server for processing. For example, this might involve continuously collecting sounds during a person's sleep for eight hours, obtaining the collected sound data, and then transmitting all the sound data to the server for comprehensive sleep sound recognition. However, in practice, it has been found that the need for cloud servers to recognize all sound data leads to excessively high computational resource consumption for sleep sound recognition. Summary of the Invention

[0004] The purpose of this application is to provide a sleep sound recognition method, device, electronic device, and storage medium to improve the problem of excessive computational resource consumption in sleep sound recognition.

[0005] This application provides a sleep sound recognition method, including: acquiring sound data and determining valid data from the sound data; performing sleep sound recognition on the valid data to obtain a sleep sound recognition result. In the implementation of the above scheme, by determining valid data from the sound data and performing sleep sound recognition on the valid data, the recognition of all sleep sounds is improved, effectively reducing the computational power consumption and computational resource consumption of sleep sound recognition.

[0006] Optionally, in this embodiment, determining valid data from the sound data includes: determining valid data from the sound data based on the average energy of the sound data. In the implementation of the above scheme, by determining only the valid data to be identified from the sound data based on the average energy of the sound data, the amount of data for sound recognition can be effectively reduced, thereby effectively reducing the computational power consumption and computational resource consumption of sleep sound recognition.

[0007] Optionally, in this embodiment, determining valid data from the sound data based on the average energy of the sound data includes: obtaining the current sound segment from the sound data and calculating the average energy of the current sound segment; dividing the current sound segment into multiple sound frames and calculating the average energy of each sound frame; for each sound frame, determining whether the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold; if so, determining the sound frame as a valid sound frame, and determining whether the current sound segment is valid data based on the valid sound frame. In the implementation of the above scheme, by determining the valid sound frame based on whether the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than the first ratio threshold, the method is not affected by the level and volume itself, thus improving the accuracy of determining valid data from the sound data.

[0008] Optionally, in this embodiment, determining valid data from the sound data based on the average energy of the sound data includes: obtaining the current sound segment from the sound data and obtaining the noise floor energy of the previous sound segment; dividing the current sound segment into multiple sound frames and calculating the average energy of each sound frame; for each sound frame, determining whether the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second ratio threshold; if so, determining the sound frame as a valid sound frame, and determining whether the current sound segment is valid data based on the valid sound frame. In the implementation of the above scheme, by using an adaptive denoising method based on time-domain signals designed according to the average energy, the determination of valid data based on the average energy is not affected by the level and volume itself, thus improving the accuracy of determining valid data from the sound data.

[0009] Optionally, in this embodiment, determining valid data from the sound data based on the average energy of the sound data includes: obtaining the current sound segment from the sound data and obtaining the noise floor energy of the previous sound segment; dividing the current sound segment into multiple sound frames and calculating the average energy of each sound frame; determining whether the sum of a first valid number and a second valid number in the multiple sound frames is greater than a preset threshold, wherein the first valid number is the number of sound frames whose ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first proportional threshold, and the second valid number is the number of sound frames whose ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second proportional threshold; if so, the current sound segment is determined as valid data. In the implementation of the above scheme, by using an adaptive denoising method based on time-domain signals designed according to the average energy, this adaptive denoising method only involves simple calculations and comparisons of the current sound segment. Therefore, this adaptive denoising method has low computational and storage requirements, thereby effectively improving the real-time performance of sleep sound recognition.

[0010] Optionally, in this embodiment, the method further includes: determining whether the previous sound segment is valid data; if so, determining the noise floor energy of the previous sound segment as the noise floor energy of the current sound segment; otherwise, determining the noise floor energy of the current sound segment based on the noise floor energy of the previous sound segment and the average energy of the current sound segment. In the implementation of the above scheme, by using an adaptive denoising method based on the time-domain signal designed according to the average energy, the determination of the noise floor energy based on the average energy is not affected by the level or volume itself, thus improving the accuracy of determining the noise floor energy from the sound data.

[0011] Optionally, in this embodiment, the method further includes deleting the audio segment preceding the previous audio segment from the audio data. In implementing the above solution, since only the current audio segment and the previous audio segment are needed to determine valid data, deleting the audio segment preceding the previous audio segment from the audio data can effectively save storage space.

[0012] Optionally, in this embodiment, sleep sound recognition of valid data includes: using a neural network model to perform sleep sound recognition on the valid data. In the implementation of the above scheme, using a lightweight neural network model to perform sleep sound recognition on the valid data reduces the number of parameters and computational complexity of the neural network model.

[0013] Optionally, in this embodiment, the neural network model includes: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separable convolutional layer; using the neural network model to perform sleep sound recognition on valid data includes: inputting valid data into the neural network model so that the neural network model can perform sleep sound recognition on the valid data. In the implementation of the above scheme, a large number of lightweight one-dimensional pointwise convolutional layers and one-dimensional depthwise separable convolutional layers are used, thereby reducing the number of parameters and computational complexity of the neural network model.

[0014] Optionally, in this embodiment, valid data is input into a neural network model to enable the model to perform sleep sound recognition on the valid data. This includes: performing convolution processing on the valid data using a one-dimensional convolutional layer of the neural network model to obtain convolutional features; normalizing the convolutional features using a normalization layer of the neural network model to obtain normalized features; downsampling the normalized features using at least one sub-network module of the neural network model to obtain downsampled features, wherein the sub-network module includes: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise segregating convolutional layer; pooling the downsampled features using a pooling layer of the neural network model to obtain pooled features; and performing fully connected operations on the pooled features using a fully connected layer of the neural network model to obtain sleep sound recognition results. In the implementation of the above scheme, by using a trained lightweight sub-network module to perform sleep sound recognition on the valid data, the number of parameters and computational complexity of the neural network model are reduced.

[0015] Optionally, in this embodiment, before inputting valid data into the neural network model to enable the neural network model to perform sleep sound recognition on the valid data, the method further includes: acquiring multiple sound sample data and multiple sound sample labels, where the sound sample labels are category labels for the sound sample data; training the neural network using the multiple sound sample data as training data and the multiple sound sample labels as training labels to obtain a neural network model; wherein, acquiring multiple sound sample data and multiple sound sample labels includes: for each sound sample data in the multiple sound sample data, determining whether the sound sample data is valid based on the average energy; if so, labeling the sound sample data to obtain the corresponding sound sample label. In the implementation of the above scheme, by using a trained lightweight neural network model to perform sleep sound recognition on valid data, the number of parameters and computational complexity of the neural network model are reduced.

[0016] Optionally, in this embodiment, the effective data input to the neural network model is one-dimensional data. In the implementation of the above scheme, only the effective data of the one-dimensional time-domain signal needs to be input into the neural network model for identification. This not only reduces the computational overhead of feature extraction but also reduces the computational process of two-dimensional feature modeling to that of one-dimensional feature modeling, thereby greatly reducing the computational complexity of the neural network model.

[0017] Optionally, in this embodiment, the sleep sound recognition result includes: multiple category probabilities of the current sound segment; after obtaining the sleep sound recognition result, it further includes: determining whether the difference between the maximum and minimum values ​​of the multiple category probabilities of the current sound segment is less than a preset threshold; if so, the current sound segment is uploaded to the server so that the server stores the current sound segment as a training dataset, or, sleep sound recognition is performed on the current sound segment to obtain the sleep sound recognition result of the current sound segment. In the implementation of the above scheme, by uploading the current sound segment to the server when the difference between the maximum and minimum values ​​of the multiple category probabilities of the current sound segment is less than a preset threshold so that the server stores the current sound segment as a training dataset, the accuracy of the neural network model in recognizing sleep sounds is improved.

[0018] Optionally, in this embodiment, after obtaining the sleep sound recognition result, the method further includes: generating a sleep quality report based on the sleep sound recognition result, or generating a health assessment report based on the sleep sound recognition result. In the implementation of the above solution, by generating a sleep quality report based on the sleep sound recognition result, or generating a health assessment report based on the sleep sound recognition result, the accuracy of the sleep quality report or health assessment report is effectively improved.

[0019] This application also provides a sleep sound recognition device, including: a valid data determination module for acquiring sound data and determining valid data from the sound data; and a recognition result acquisition module for performing sleep sound recognition on the valid data to obtain a sleep sound recognition result. In the implementation of the above solution, by determining valid data from the sound data and performing sleep sound recognition on the valid data, the recognition of all sleep sounds is improved, effectively reducing the computational power consumption and computational resource consumption of sleep sound recognition.

[0020] Optionally, in this embodiment of the application, the valid data determination module includes: a valid data determination submodule, used to determine valid data from the sound data based on the average energy of the sound data.

[0021] Optionally, in this embodiment, the valid data determination submodule includes: a first sound segment calculation unit, configured to obtain the current sound segment from the sound data and calculate the average energy of the current sound segment; a first average energy calculation unit, configured to divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame in the multiple sound frames; a first average energy judgment unit, configured to determine whether the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold for each sound frame in the multiple sound frames; and a first valid data determination unit, configured to determine the sound frame as a valid sound frame if the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than the first ratio threshold, and determine whether the current sound segment is valid data based on the valid sound frame.

[0022] Optionally, in this embodiment, the valid data determination submodule includes: a second sound segment calculation unit, configured to obtain the current sound segment from the sound data and obtain the noise floor energy of the previous sound segment; a second average energy calculation unit, configured to divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame in the multiple sound frames; a second average energy judgment unit, configured to determine whether the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second ratio threshold for each sound frame in the multiple sound frames; and a second valid data determination unit, configured to determine the sound frame as a valid sound frame if the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than the second ratio threshold, and determine whether the current sound segment is valid data based on the valid sound frame.

[0023] Optionally, in this embodiment, the valid data determination submodule includes: a third sound segment calculation unit, configured to obtain the current sound segment and the previous sound segment from the sound data, and calculate the average energy and background noise energy of the previous sound segment; a third average energy calculation unit, configured to divide the current sound segment into multiple sound frames, and calculate the average energy of each sound frame in the multiple sound frames; a third average energy judgment unit, configured to judge whether the sum of the first valid quantity and the second valid quantity in the multiple sound frames is greater than a preset quantity threshold, wherein the first valid quantity is the number of sound frames whose ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first proportional threshold, and the second valid quantity is the number of sound frames whose ratio between the average energy of the sound frame and the background noise energy of the previous sound segment is greater than a second proportional threshold; and a third valid data determination unit, configured to determine the current sound segment as valid data if the sum of the first valid quantity and the second valid quantity in the multiple sound frames is greater than the preset quantity threshold.

[0024] Optionally, in this embodiment of the application, the sleep sound recognition device further includes: a sound segment judgment module, used to judge whether the previous sound segment is valid data; and a noise floor energy determination module, used to determine the noise floor energy of the previous sound segment as the noise floor energy of the current sound segment if the previous sound segment is valid data, otherwise, to determine the noise floor energy of the current sound segment based on the noise floor energy of the previous sound segment and the average energy of the current sound segment.

[0025] Optionally, in this embodiment of the application, the sleep sound recognition device further includes: a sound segment deletion module, used to delete the sound segment preceding the previous sound segment from the sound data.

[0026] Optionally, in this embodiment of the application, the recognition result acquisition module includes: a sleep sound recognition submodule, used to perform sleep sound recognition on valid data using a neural network model.

[0027] Optionally, in this embodiment, the neural network model includes: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separable convolutional layer; the sleep sound recognition submodule includes: an effective data input unit, used to input effective data into the neural network model so that the neural network model can perform sleep sound recognition on the effective data.

[0028] Optionally, in this embodiment, the effective data input unit includes: a convolutional feature acquisition subunit, used to perform convolution processing on the effective data using a one-dimensional convolutional layer of a neural network model to obtain convolutional features; a feature normalization processing subunit, used to perform normalization processing on the convolutional features using a normalization layer of a neural network model to obtain normalized features; a feature downsampling processing unit, used to perform downsampling processing on the normalized features using at least one sub-network module of a neural network model to obtain downsampling features, the sub-network module including: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise segregating convolutional layer; a pooling feature acquisition subunit, used to perform pooling processing on the downsampling features using a pooling layer of a neural network model to obtain pooling features; and a recognition result acquisition subunit, used to perform fully connected operations on the pooling features using a fully connected layer of a neural network model to obtain sleep sound recognition results.

[0029] Optionally, in this embodiment of the application, the sleep sound recognition device further includes: a data label acquisition module, used to acquire multiple sound sample data and multiple sound sample labels, wherein the sound sample labels are category labels of the sound sample data; and a network model training module, used to train a neural network using the multiple sound sample data as training data and the multiple sound sample labels as training labels to obtain a neural network model; wherein acquiring multiple sound sample data and multiple sound sample labels includes: for each sound sample data in the multiple sound sample data, determining whether the sound sample data is valid based on the average energy; if so, labeling the sound sample data to obtain the sound sample label corresponding to the sound sample data.

[0030] Optionally, in this embodiment of the application, the effective data input to the neural network model is one-dimensional data.

[0031] Optionally, in this embodiment, the sleep sound recognition result includes: multiple category probabilities of the current sound segment; the sleep sound recognition device further includes: a category probability judgment module, used to determine whether the difference between the maximum and minimum values ​​of the multiple category probabilities of the current sound segment is less than a preset threshold; and a sound segment uploading module, used to upload the current sound segment to the server if the difference between the maximum and minimum values ​​of the multiple category probabilities of the current sound segment is less than the preset threshold, so that the server stores the current sound segment as a training dataset, or to perform sleep sound recognition on the current sound segment to obtain the sleep sound recognition result of the current sound segment.

[0032] Optionally, in this embodiment of the application, the sleep sound recognition device further includes: a result report generation module, used to generate a sleep quality report based on the sleep sound recognition result, or to generate a health assessment report based on the sleep sound recognition result.

[0033] This application also provides an electronic device, including a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions, when executed by the processor, perform the method described above.

[0034] Optionally, in embodiments of this application, the electronic device includes: a mobile device, a wearable device, a home appliance, or a medical device.

[0035] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to perform the methods described above.

[0036] Other features and advantages of embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of this application. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 The diagram shown is a flowchart of the sleep sound recognition method provided in an embodiment of this application;

[0039] Figure 2 The diagram shown is a structural schematic of the neural network model provided in an embodiment of this application;

[0040] Figure 3 The diagram shown is a schematic representation of the network structure of the sub-network module provided in an embodiment of this application.

[0041] Figure 4 The diagram shown is a flowchart illustrating the training process of a neural network model provided in an embodiment of this application.

[0042] Figure 5 The diagram shown is a structural schematic of the sleep sound recognition device provided in an embodiment of this application;

[0043] Figure 6 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of this application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0045] It is understood that the terms "first" and "second" in the embodiments of this application are used to distinguish similar objects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0046] Before introducing the sleep sound recognition method provided in the embodiments of this application, let's first introduce some concepts involved in the embodiments of this application:

[0047] Artificial intelligence (AI) is a new field of technical science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems.

[0048] It should be noted that the sleep sound recognition method provided in this application embodiment can be executed by an electronic device. Here, an electronic device refers to a device terminal or server with the function of executing computer programs. Device terminals can be divided into four main categories: mobile devices, wearable devices, home appliances, and medical devices. Mobile devices include, for example, smartphones, tablets, personal digital assistants, or mobile internet devices; wearable devices include, for example, smart bracelets, smartwatches, sleep monitoring armbands, and smart glasses; home appliances include, for example, televisions, desktop computers, air conditioners, refrigerators, smart mattresses, smart electronic beds, and smart speakers; and medical devices include, for example, electrocardiogram (ECG) instruments, fetal monitoring instruments, and sleep aids. It is understood that the sleep sound recognition method can include two stages: a first stage, determining valid data from sound data; and a second stage, performing sleep sound recognition on the valid data. When the first and second stages are executed on different electronic devices, there are three different scenarios:

[0049] In the first scenario, both the first and second stages are executed on the terminal device. For example, the terminal device determines valid data from the sound data and performs sleep sound recognition on the valid data to obtain the sleep sound recognition result.

[0050] In the second scenario, the first stage is executed on the terminal device, while the second stage is executed on the server. Specifically, the terminal device identifies valid data from the audio data and sends this valid data to the server. The server receives the valid data from the terminal device, performs sleep sound recognition on the valid data, and obtains the sleep sound recognition result.

[0051] In the third scenario, both the first and second stages are executed on the server. For example, the server determines valid data from the sound data and performs sleep sound recognition on the valid data to obtain the sleep sound recognition result.

[0052] The following describes the applicable scenarios for this sleep sound recognition method. These scenarios include, but are not limited to, using this method to enhance or improve the functionality of sleep devices, such as smart bracelets, smartphones, sleep monitors, smart mattresses, smart electronic beds, smart speakers, ECG devices, and sleep aids. In practical applications, this method can also be used to identify valid data from sound data, allowing electronic devices to perform sleep sound recognition only on valid data. This improves the recognition of all sleep sounds, thereby reducing the computational power consumption and resource consumption of the electronic device. Furthermore, after identifying valid data from the sound data using this method, the electronic device can send only the valid data to the server, avoiding the transmission of all data. This allows the server to perform sleep sound recognition only on the valid data, improving the recognition of all sleep sounds and further reducing the server's computational power consumption and resource consumption.

[0053] Please see Figure 1 The illustration shows a flowchart of a sleep sound recognition method provided in an embodiment of this application; this application provides a sleep sound recognition method, including:

[0054] Step S110: Acquire sound data and determine valid data from the sound data.

[0055] It is understood that the methods for acquiring the aforementioned sound data include, but are not limited to: the first method, using sleep monitoring devices such as sleep monitors, smart mattresses, smart electronic beds, smart bracelets, and smartphones to collect sound data from the target object; then the sleep device sends the sound data to an electronic device, which receives the sound data sent by the sleep device, and the electronic device can store the sound data in a file system, database, or mobile storage device; the second method, acquiring pre-stored sound data, specifically, for example, acquiring sound data from a file system, database, or mobile storage device; the third method, using software such as browsers to acquire sound data from the Internet, or using other applications to access the Internet to acquire sound data.

[0056] The specific method for collecting the above sound data is as follows: use the built-in microphone of the mobile phone to record sound data, and assume the sampling frequency is A Hz (e.g., 8000 Hz, 12000 Hz or 16000 Hz, etc.). After buffering the sound data for t seconds (e.g., 3 seconds, 4 seconds or 5 seconds, etc.) as a sound segment, sleep recognition can be performed on the sound data of this t-second sound segment in real time.

[0057] Step S120: Perform sleep sound recognition on the valid data to obtain sleep sound recognition results.

[0058] Sleep sound recognition results refer to the identification of sound categories collected during sleep. These sound categories include, but are not limited to: snoring, sleep talking, breathing, turning over, teeth grinding, coughing, etc., as well as environmental sounds during sleep, such as cat, dog, bird calls, baby crying, car horns, rain sounds, etc. It is understood that there are many ways to implement steps S110 to S120 above; therefore, the implementation methods of steps S110 to S120 will be described in detail below.

[0059] In the above implementation process, by identifying valid data from the sound data and performing sleep sound recognition on the valid data, the recognition of all sleep sounds is improved, and the computational power consumption and computational resource consumption of sleep sound recognition are effectively reduced.

[0060] As a first optional implementation of step S110 above, when determining valid data, the valid data can be determined based on the average energy. This implementation may include:

[0061] Step S111: Determine valid data from the sound data based on the average energy of the sound data.

[0062] Understandably, the purpose of determining valid data from sound data based on its average energy is to filter out background noise generated during recording (i.e., noise produced by the electronic device itself even in a quiet environment) or stable environmental noise (such as the low, steady hum of machinery in a nearby factory). This noise is useless for sleep recognition, and to avoid interference with the results, it can be excluded from the sound data; that is, only valid data needs to be determined.

[0063] In practice, it was found that there is a significant difference in energy fluctuation between noise data and valid data. Therefore, the average energy can be used to distinguish between noise data and valid data. According to statistics, noise accounts for 50% to 80% of the sound data during a night's sleep. Therefore, identifying only the valid data that needs to be recognized from the sound data can effectively reduce the amount of data for sound recognition, thereby effectively reducing the computational power consumption and computational resource consumption of sleep sound recognition.

[0064] As a first optional implementation of step S111 above, when determining valid data based on average energy, valid data can be determined based on the ratio between short-time average energy (i.e., the average energy of a sound frame) and long-time average energy (i.e., the average energy of a sound segment). This implementation may include:

[0065] Step S111a: Obtain the current sound segment from the sound data and calculate the average energy of the current sound segment.

[0066] Step S111b: Divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame in the multiple sound frames.

[0067] The implementation of steps S111a to S111b above is as follows: Assuming that A Hz sound per t seconds is considered as one processed sound segment, then the sound data of the entire night's sleep can be divided into m sound segments, and each sound segment has t×A sample values. Therefore, each sound segment can be divided into t×q sound frames, and each sound frame has A / q sample values. It is understood that q here can be set according to specific circumstances, specifically 5, 8, 10, 12, 14, 16, 18, or 20, etc. For ease of understanding, calculation, and description, the following description uses q as an example of 10. Assuming the current sound segment is represented by the k-th sound segment, the n-th sample value of the l-th sound frame of the k-th sound segment can be represented by the formula f(k,l,n), k=0,1,2…m,l=0,1,2…t*10. To represent, and then use the formula Calculate the average energy of each sound frame; where e(k,l) represents the average energy of the l-th sound frame of the k-th sound segment, and A represents the sampling frequency.

[0068] Step S111c: For each of the multiple sound frames, determine whether the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold.

[0069] Step S111d: If the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than the first ratio threshold, then the sound frame is determined as a valid sound frame, and the current sound segment is determined as valid data based on the valid sound frame.

[0070] For example, the implementation of steps S111c to S111d above can be achieved using the formula: Calculate the number of valid audio frames in the current audio segment, where H e (k) represents the number of valid sound frames in the current sound segment (i.e., the number of sound frames whose average energy is greater than the first proportional threshold compared to the average energy of the current sound segment), h e (k,l) indicates whether the l-th sound frame of the k-th sound segment is a valid sound frame. Here, a valid sound frame is defined as a sound frame whose average energy is greater than the average energy of the current sound segment.

[0071] Assuming the first ratio threshold is set to 4.4, in practice, the first ratio threshold can be set according to specific circumstances. Therefore, whether the aforementioned audio frame is a valid audio frame can be determined using the formula... To determine, where h e (k,l) indicates whether the l-th sound frame of the k-th sound segment is a valid sound frame, where 1 indicates a valid sound frame and 0 indicates a invalid sound frame. e(k,l) represents the average energy of the l-th sound frame of the k-th sound segment, and E(k) represents the average energy of the current sound segment, which is the average energy of the k-th sound segment.

[0072] In practical application, it can be determined whether the number of valid sound frames in the current sound segment exceeds a preset threshold. This preset threshold can be set according to specific circumstances, such as 5% or 10% of the total number of sound frames in the sound segment. Specifically, if the current sound segment is divided into t×... q If there are 10 sound frames, the preset quantity threshold is set to t×q×5% or t×q×10%, etc. Here, a valid sound frame is defined as a sound frame whose average energy is greater than the average energy of the current sound segment. If the number of valid sound frames in the current sound segment is greater than the preset quantity threshold, the current sound segment is determined to be valid data; similarly, if the number of valid sound frames in the current sound segment is less than or equal to the preset quantity threshold, the current sound segment is determined to be invalid data (i.e., noise data).

[0073] As a second optional implementation of step S111 above, when determining valid data based on average energy, the valid data can be determined based on the ratio between short-time average energy (i.e., the average energy of the sound frame) and background noise energy. This implementation may include:

[0074] Step S111e: Obtain the current sound segment from the sound data and obtain the noise floor energy of the previous sound segment.

[0075] Step S111f: Divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame in the multiple sound frames.

[0076] The implementation principles and methods of steps S111e to S111f are similar to those of steps S111a to S111b. Therefore, their implementation principles and methods will not be described here. If there is anything unclear, please refer to the description of steps S111a to S111b.

[0077] Step S111g: For each of the multiple sound frames, determine whether the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than the second ratio threshold.

[0078] Step S111h: If the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than the second ratio threshold, then the sound frame is determined as a valid sound frame, and the current sound segment is determined as valid data based on the valid sound frame.

[0079] For example, the implementation of steps S111g to S111h above can be achieved using the formula: Calculate the number of valid audio frames in the current audio segment, where H em (k) The number of valid sound frames in the current sound segment. Here, a valid sound frame is defined as a sound frame whose average energy is greater than the noise floor energy of the previous sound segment, and the ratio is greater than the second proportional threshold. em (k,l) indicates whether the l-th sound frame of the k-th sound segment is a valid sound frame. Here, a valid sound frame refers to whether the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than the second ratio threshold.

[0080] Assuming the second ratio threshold is set to 3.7, in practice, the second ratio threshold can be set according to specific circumstances. Therefore, whether the aforementioned audio frame is a valid audio frame can be determined using the formula... To determine, where h em (k,l) indicates whether the l-th sound frame of the k-th sound segment is a valid sound frame, where 1 indicates a valid sound frame and 0 indicates a invalid sound frame. e(k,l) represents the average energy of the l-th sound frame of the k-th sound segment, and EM(k-1) represents the noise floor energy of the previous sound segment, which is the noise floor energy of the (k-1)-th sound segment.

[0081] In practice, it can be determined whether the number of valid sound frames in the current sound segment is greater than a preset threshold. This preset threshold can be set according to specific circumstances, such as 5% or 10% of the total number of sound frames in the sound segment. Specifically, if the current sound segment is divided into t×q sound frames, the preset threshold could be set to t×q×5% or t×q×10%, etc. A valid sound frame is defined as a sound frame whose average energy is greater than the noise floor energy of the previous sound segment, and the ratio is greater than a second proportional threshold. If the number of valid sound frames in the current sound segment is greater than the preset threshold, the current sound segment is determined to be valid data. Similarly, if the number of valid sound frames in the current sound segment is less than or equal to the preset threshold, the current sound segment is determined to be invalid data (i.e., noise data).

[0082] In the above implementation process, by using an adaptive denoising method based on time-domain signals designed according to average energy, the determination of effective data based on average energy is not affected by the level and volume itself, thus improving the accuracy of determining effective data from sound data.

[0083] As a third optional implementation of step S111 above, when determining valid data based on average energy, the above two implementations can be combined, and the ratio between short-time average energy and long-time average energy, as well as the ratio between short-time average energy and noise floor energy, can be combined to determine valid data. This implementation may include:

[0084] Step S111i: Obtain the current sound segment from the sound data and obtain the noise floor energy of the previous sound segment.

[0085] Step S111j: Divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame in the multiple sound frames.

[0086] The implementation principles and methods of steps S111i to S111j are similar to those of steps S111a to S111b. Therefore, their implementation principles and methods will not be described here. If there is anything unclear, please refer to the description of steps S111a to S111b.

[0087] Step S111k: Determine whether the sum of the first effective quantity and the second effective quantity in multiple sound frames is greater than a preset quantity threshold. The first effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold. The second effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second ratio threshold.

[0088] Step S111m: If the sum of the first and second valid quantities in multiple sound frames is greater than a preset quantity threshold, then the current sound segment is determined as valid data.

[0089] For example, the implementation of steps S111k to S111m above can be achieved using the formula: Determine whether the current audio segment is valid data; where V(k) represents whether the current audio segment is valid data, 1 indicates a valid audio frame, 0 indicates a invalid audio frame, and H... e (k) represents the first effective quantity, which is the number of sound frames whose ratio of the average energy of the sound frame to the average energy of the current sound segment is greater than a first ratio threshold, H. em (k) represents the second effective quantity, which is the number of sound frames whose ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than the second proportional threshold. γ represents the preset quantity threshold, which can be set according to specific circumstances. For example, if the current sound segment is divided into t×q sound frames, the preset quantity threshold γ can be set to t×q×5% or t×q×10%, etc.

[0090] In the implementation of the above scheme, an adaptive denoising method based on time-domain signals is designed according to the average energy. This adaptive denoising method only involves simple calculation and comparison of the current sound segment. Therefore, this adaptive denoising method has low requirements for computation and storage, thereby effectively improving the real-time performance of sleep sound recognition.

[0091] As an optional implementation of the above-mentioned sleep sound recognition method, the background noise energy can also be updated. This implementation may include:

[0092] Step S112: Determine whether the previous audio segment is valid data.

[0093] Step S113: If the previous sound segment is valid data, then the noise floor energy of the previous sound segment is determined as the noise floor energy of the current sound segment.

[0094] Step S114: If the previous sound segment is not valid data, determine the noise floor energy of the current sound segment based on the noise floor energy of the previous sound segment and the average energy of the current sound segment.

[0095] For example, the implementation of steps S112 to S114 above can be achieved using the following formula. The system determines whether the previous sound segment is valid data and determines the noise floor energy of the current sound segment based on the determination result. Here, EM(k) represents the noise floor energy of the current sound segment, i.e., the noise floor energy of the kth sound segment; E(k) represents the average energy of the current sound segment, i.e., the average energy of the kth sound segment; EM(k-1) represents the noise floor energy of the previous sound segment, i.e., the noise floor energy of the (k-1)th sound segment; α is a smoothing coefficient, which can be set according to specific circumstances, for example, the smoothing coefficient can be set to 0.04; V(k) indicates whether the current sound segment is valid data, 1 indicates that it is a valid sound frame, and 0 indicates that it is not a valid sound frame.

[0096] In the above implementation process, by using an adaptive noise reduction method based on time-domain signals designed according to the average energy, the noise floor energy is not affected by the level volume itself when determining the noise floor energy based on the average energy, thus improving the accuracy of determining the noise floor energy from the sound data.

[0097] As an optional implementation of the aforementioned sleep sound recognition method, sound segments preceding the previous sound segment can be deleted to save storage space. This implementation may include:

[0098] Step S115: Delete the audio segment preceding the previous audio segment from the audio data.

[0099] An example implementation of step S115 above involves using an executable program compiled or interpreted in a preset programming language to store the current sound segment and the previous sound segment in a cache database such as Memcached and Redis, and deleting the sound segment preceding the previous sound segment from the sound data stored in the cache database. The programming languages ​​that can be used include, for example, C, C++, Java, BASIC, JavaScript, LISP, Shell, Perl, Ruby, Python, and PHP. In the implementation of the above scheme, since only the current sound segment and the previous sound segment are needed to determine the valid data, deleting the sound segment preceding the previous sound segment from the sound data can effectively save storage space. Optionally, after determining the noise floor energy of the current sound segment, the previous sound segment can also be deleted, i.e., only the current sound segment needs to be retained, thereby further saving storage space.

[0100] As a second optional implementation of step S110 above, other network models can also be used to determine valid data. Specifically, this implementation includes using a Convolutional Neural Network (CNN) model or a Recurrent Neural Network (RNN) model to determine valid data from the audio data. Examples of CNN models that can be used include LeNet, AlexNet, VGG, GoogLeNet, and ResNet, etc. Examples of RNN models that can be used include Long Short-Term Memory (LSTM) networks and Bidirectional Long Short-Term Memory (Bi-LSTM) networks, etc.

[0101] As an optional implementation of step S120 above, when performing sleep sound recognition on valid data, a neural network model can be used for recognition. This implementation may include:

[0102] Step S121: Use a neural network model to identify sleep sounds from valid data.

[0103] Please see Figure 2 The diagram shows a schematic of the neural network model provided in this application embodiment. The neural network model refers to a trained neural network model that performs sleep sound recognition on valid data. This neural network model can be a self-constructed model. Here, a self-constructed neural network model is used as an example. This neural network model may include: a one-dimensional convolutional layer (Conv1d), a normalization layer (Norm), multiple sub-network modules (SubModel), a pooling layer (Pool), and a fully connected layer (Linear), etc. The one-dimensional convolutional layer (Conv1d), normalization layer (Norm), multiple sub-network modules (SubModel), pooling layer (Pool), and fully connected layer (Linear) are connected sequentially. The multiple sub-network modules (SubModel) may include: sub-network module 1 (SubModel-1), sub-network module 2 (SubModel-2), ..., sub-network module n (SubModel-n), and the multiple sub-network modules (SubModel) are also connected sequentially.

[0104] As an optional implementation of step S121 above, the neural network model may include: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separating convolutional layer; using the neural network model to perform sleep sound recognition on valid data includes:

[0105] Step S122: Input the valid data into the neural network model so that the neural network model can recognize sleep sounds from the valid data.

[0106] Please see Figure 3 The diagram shows a schematic of the network structure of a sub-network module provided in an embodiment of this application. The aforementioned one-dimensional pointwise convolutional layer and one-dimensional depthwise separable convolutional layer can be placed in the sub-network module, or they can be omitted. For ease of understanding and explanation, this is illustrated by taking the placement of the one-dimensional pointwise convolutional layer and one-dimensional depthwise separable convolutional layer in the sub-network module as an example. The aforementioned sub-network module may include: a one-dimensional pointwise convolutional layer, a one-dimensional depthwise separable convolutional layer, a normalization layer, an activation function, etc., wherein the connection relationships of the various layers in the aforementioned sub-network module are as follows: Figure 3 As shown, it will not be elaborated further.

[0107] It is understandable that a one-dimensional pointwise convolutional layer with m input channels and n output channels can be represented as PointConv1d(m, n), and a one-dimensional depthwise segregating convolutional layer with m input channels, n output channels, a convolution size of k, and a stride of s can be represented as Deep Conv1d(m, n, k, s). Similarly, a one-dimensional depthwise segregating convolutional layer with n input channels, n output channels, a convolution size of k, and a stride of s can be represented as Deep Conv1d(n, n, k, s). The one-dimensional depthwise segregating convolutional layer has a downsampling function; in practice, the sampling factor can be adjusted by changing the convolution stride. In the implementation of the above scheme, a large number of lightweight one-dimensional pointwise convolutional layers and one-dimensional depthwise segregating convolutional layers are used, thereby reducing the number of parameters and computational complexity of the neural network model.

[0108] As an optional implementation of step S122 above, the implementation of inputting effective data into the neural network model may include:

[0109] Step S122a: Use a one-dimensional convolutional layer of the neural network model to perform convolution processing on the effective data to obtain convolutional features.

[0110] Step S122b: Use the normalization layer of the neural network model to normalize the convolutional features to obtain normalized features.

[0111] Step S122c: Use at least one sub-network module of the neural network model to downsample the normalized features to obtain downsampled features. The sub-network module includes: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separating convolutional layer.

[0112] Step S122d: Use the pooling layer of the neural network model to perform pooling processing on the downsampled features to obtain pooled features.

[0113] Step S122e: Use the fully connected layer of the neural network model to perform fully connected operations on the pooled features to obtain the sleep sound recognition results.

[0114] It is understandable that the sleep sound recognition result output by the fully connected layer of the above neural network model can be multiple category probabilities. Therefore, the category with the highest probability among the multiple category probabilities can be determined as the sleep sound recognition result. For example, assuming that the probability of these multiple category probabilities is: snoring is 10%, sleep talking is 70%, breathing is 15%, and the sum of the probabilities of other categories is 5%, then it can be confirmed that the final sleep sound recognition result is sleep talking.

[0115] Please see Figure 4 The illustrated flowchart shows a training neural network model provided in an embodiment of this application. As an optional implementation of the above-described sleep sound recognition method, before inputting valid data into the neural network model to enable the neural network model to perform sleep sound recognition on the valid data, the neural network model can be trained. This implementation may include:

[0116] Step S210: Obtain multiple sound sample data and multiple sound sample labels, where the sound sample labels are the category labels of the sound sample data.

[0117] Step S220: Train the neural network using multiple sound sample data as training data and multiple sound sample labels as training labels to obtain a neural network model.

[0118] The process of acquiring multiple sound sample data and multiple sound sample labels includes: for each sound sample data in the multiple sound sample data, determining whether the sound sample data is valid based on the average energy; if so, labeling the sound sample data to obtain the corresponding sound sample label.

[0119] For example, the implementation of steps S210 to S220 above involves: acquiring multiple sound sample data and multiple sound sample labels, where the sound sample labels are the category labels of the sound sample data; using a neural network to predict the sound sample data to obtain the predicted category; calculating the loss value between the predicted category and the category label in the sound sample labels; and training the neural network based on the loss value until the loss value is less than a preset threshold, thus obtaining the neural network model.

[0120] In the above implementation process, after determining whether the sound sample data is valid based on the average energy, the sound sample data is labeled, thereby effectively reducing the labeling time of the sound sample data and thus effectively improving the training speed of the neural network model.

[0121] As an alternative implementation of the aforementioned sleep sound recognition method, the effective data input to the neural network model can be one-dimensional data. Compared to the traditional method of using Short-Time Fourier Transform (STFT) for feature extraction and converting it into a two-dimensional spectrogram signal, and then using an AI model to recognize the two-dimensional spectrogram signal, the above scheme only requires inputting the effective data of the one-dimensional time-domain signal into the neural network model for recognition. This not only reduces the computational overhead of feature extraction but also reduces the computational process of two-dimensional feature modeling to one-dimensional feature modeling, thereby greatly reducing the computational complexity of the neural network model.

[0122] As an optional implementation of the above-described sleep sound recognition method, the sleep sound recognition result may include: multiple category probabilities of the current sound segment; after obtaining the sleep sound recognition result, sound segments whose categories are difficult to determine may also be uploaded to the server, and this implementation may include:

[0123] Step S230: Determine whether the difference between the maximum and minimum probabilities of the current sound segment among multiple categories is less than a preset threshold.

[0124] An example implementation of step S230 above involves using an executable program compiled or interpreted in a preset programming language to determine whether the difference between the maximum and minimum probabilities of multiple categories in the current sound segment is less than a preset threshold. The programming languages ​​that can be used include, for example, C, C++, Java, BASIC, JavaScript, LISP, Shell, Perl, Ruby, Python, and PHP. The preset threshold refers to a pre-set limit threshold, which can be set according to specific circumstances, for example, it can be set to 2, 5, or 9.

[0125] Step S240: If the difference between the maximum and minimum probabilities of multiple categories of the current sound segment is less than a preset threshold, then the current sound segment is uploaded to the server so that the server stores the current sound segment as a training dataset, or sleep sound recognition is performed on the current sound segment to obtain the sleep sound recognition result of the current sound segment.

[0126] For example, in implementing step S240 above: assuming the preset threshold is 5%, and the probabilities of these multiple categories are: 33% for snoring, 35% for sleep talking, and 32% for breathing, then it is clear that the maximum value of the probabilities of these three categories (snoring, sleep talking, and breathing) is 35%, and the minimum value is 32%. The difference between the maximum and minimum values, 3%, is less than the preset threshold of 5%. Therefore, the current sound segment can be uploaded to the server so that the server can store the current sound segment as a training dataset, or the current sound segment can be uploaded to the server so that the server can perform sleep sound recognition on the current sound segment and obtain the sleep sound recognition result of the current sound segment.

[0127] In the implementation of the above scheme, when the difference between the maximum and minimum values ​​among the multiple category probabilities of the current sound segment is less than a preset threshold, the current sound segment is uploaded to the server so that the server stores the current sound segment as a training dataset, thereby improving the accuracy of the neural network model in recognizing sleep sounds.

[0128] As an optional implementation of the above-mentioned sleep sound recognition method, after obtaining the sleep sound recognition result, it may also generate:

[0129] Step S250: Generate a sleep quality report based on the sleep sound recognition results, or generate a health assessment report based on the sleep sound recognition results.

[0130] An example implementation of step S250 above involves using an executable program compiled or interpreted in a preset programming language to generate a sleep quality report or a health assessment report based on the sleep sound recognition results. The programming languages ​​that can be used include, for example, C, C++, Java, BASIC, JavaScript, LISP, Shell, Perl, Ruby, Python, and PHP. In implementing the above solution, by generating a sleep quality report or a health assessment report based on the sleep sound recognition results, the accuracy of the sleep quality report or health assessment report is effectively improved.

[0131] Please see Figure 5 The diagram shown is a structural schematic of the sleep sound recognition device provided in an embodiment of this application; this application provides a sleep sound recognition device 300, including:

[0132] The valid data determination module 310 is used to acquire sound data and determine valid data from the sound data.

[0133] The recognition result acquisition module 320 is used to perform sleep sound recognition on valid data and obtain sleep sound recognition results.

[0134] Optionally, in this embodiment of the application, the valid data determination module includes:

[0135] The valid data determination submodule is used to determine valid data from the sound data based on the average energy of the sound data.

[0136] Optionally, in this embodiment of the application, the valid data determination submodule includes:

[0137] The first sound segment calculation unit is used to obtain the current sound segment from the sound data and calculate the average energy of the current sound segment.

[0138] The first average energy calculation unit is used to divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame in the multiple sound frames.

[0139] The first average energy judgment unit is used to determine, for each of the multiple sound frames, whether the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold.

[0140] The first valid data determination unit is used to determine the sound frame as a valid sound frame if the ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold, and to determine whether the current sound segment is valid data based on the valid sound frame.

[0141] Optionally, in this embodiment of the application, the valid data determination submodule includes:

[0142] The second sound segment calculation unit is used to obtain the current sound segment from the sound data and to obtain the noise floor energy of the previous sound segment.

[0143] The second average energy calculation unit is used to divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame.

[0144] The second average energy judgment unit is used to determine, for each of the multiple sound frames, whether the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second ratio threshold.

[0145] The second valid data determination unit is used to determine the sound frame as a valid sound frame if the ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second ratio threshold, and to determine whether the current sound segment is valid data based on the valid sound frame.

[0146] Optionally, in this embodiment of the application, the valid data determination submodule includes:

[0147] The third sound segment calculation unit is used to obtain the current sound segment from the sound data and to obtain the noise floor energy of the previous sound segment.

[0148] The third average energy calculation unit is used to divide the current sound segment into multiple sound frames and calculate the average energy of each sound frame in the multiple sound frames.

[0149] The third average energy judgment unit is used to determine whether the sum of the first effective quantity and the second effective quantity in multiple sound frames is greater than a preset quantity threshold. The first effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold. The second effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the background noise energy of the previous sound segment is greater than a second ratio threshold.

[0150] The third valid data determination unit is used to determine the current sound segment as valid data if the sum of the first and second valid quantities in multiple sound frames is greater than a preset quantity threshold.

[0151] Optionally, in this embodiment of the application, the sleep sound recognition device further includes:

[0152] The audio segment determination module is used to determine whether the previous audio segment is valid data.

[0153] The noise floor energy determination module is used to determine the noise floor energy of the previous sound segment as the noise floor energy of the current sound segment if the previous sound segment is valid data; otherwise, it determines the noise floor energy of the current sound segment based on the noise floor energy of the previous sound segment and the average energy of the current sound segment.

[0154] Optionally, in this embodiment of the application, the sleep sound recognition device further includes:

[0155] The audio segment deletion module is used to delete audio segments from the audio data that precede the previous audio segment.

[0156] Optionally, in this embodiment of the application, the identification result acquisition module includes:

[0157] The sleep sound recognition submodule is used to recognize sleep sounds from valid data using a neural network model.

[0158] Optionally, in this embodiment of the application, the neural network model includes: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separating convolutional layer; the sleep sound recognition submodule includes:

[0159] The valid data input unit is used to input valid data into the neural network model so that the neural network model can perform sleep sound recognition on the valid data.

[0160] Optionally, in this embodiment of the application, the effective data input unit includes:

[0161] The convolutional feature acquisition subunit is used to perform convolution processing on effective data using the one-dimensional convolutional layer of the neural network model to obtain convolutional features.

[0162] The feature normalization processing subunit is used to normalize the convolutional features using the normalization layer of the neural network model to obtain normalized features.

[0163] The feature downsampling processing unit is used to downsample normalized features using at least one sub-network module of the neural network model to obtain downsampled features. The sub-network module includes: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separating convolutional layer.

[0164] The pooling feature acquisition subunit is used to perform pooling processing on downsampled features using the pooling layer of the neural network model to obtain pooled features.

[0165] The recognition result obtains a sub-unit, which is used to perform fully connected operations on the pooled features using the fully connected layer of the neural network model to obtain the sleep sound recognition result.

[0166] Optionally, in this embodiment of the application, the sleep sound recognition device further includes:

[0167] The data label acquisition module is used to acquire multiple sound sample data and multiple sound sample labels. The sound sample labels are the category labels of the sound sample data.

[0168] The network model training module is used to train the neural network using multiple sound sample data as training data and multiple sound sample labels as training labels to obtain a neural network model.

[0169] The process of acquiring multiple sound sample data and multiple sound sample labels includes: for each sound sample data in the multiple sound sample data, determining whether the sound sample data is valid based on the average energy; if so, labeling the sound sample data to obtain the corresponding sound sample label.

[0170] Optionally, in this embodiment of the application, the effective data input to the neural network model is one-dimensional data.

[0171] Optionally, in this embodiment, the sleep sound recognition result includes: multiple category probabilities of the current sound segment; the sleep sound recognition device further includes:

[0172] The category probability determination module is used to determine whether the difference between the maximum and minimum values ​​of the multiple category probabilities of the current sound segment is less than a preset threshold.

[0173] The audio segment upload module is used to upload the current audio segment to the server if the difference between the maximum and minimum probabilities of multiple categories of the current audio segment is less than a preset threshold, so that the server can store the current audio segment as a training dataset, or to perform sleep sound recognition on the current audio segment to obtain the sleep sound recognition result of the current audio segment.

[0174] Optionally, in this embodiment of the application, the sleep sound recognition device further includes:

[0175] The results report generation module is used to generate a sleep quality report or a health assessment report based on the sleep sound recognition results.

[0176] It should be understood that this device corresponds to the sleep sound recognition method embodiment described above and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0177] Please see Figure 6 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 400 provided in this application includes a processor 410 and a memory 420. The memory 420 stores machine-readable instructions executable by the processor 410. When the machine-readable instructions are executed by the processor 410, the method described above is performed.

[0178] This application embodiment also provides a computer-readable storage medium 430, on which a computer program is stored. This computer program is executed by a processor 410 to perform the methods described above. The computer-readable storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0179] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0180] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending primarily on the functions involved.

[0181] Furthermore, the functional modules of each embodiment in this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0182] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A sleep sound recognition method, characterized in that, include: Acquire sound data; Obtain the current sound segment from the sound data, and obtain the noise floor energy of the previous sound segment; The current sound segment is divided into multiple sound frames, and the average energy of each sound frame in the multiple sound frames is calculated; Determine whether the sum of the first effective quantity and the second effective quantity in the plurality of sound frames is greater than a preset quantity threshold. The first effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold. The second effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second ratio threshold. If so, the current audio segment is determined as valid data; The effective data is used to identify sleep sounds using a neural network model to obtain sleep sound recognition results. The neural network model includes a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separating convolutional layer.

2. The method according to claim 1, characterized in that, Also includes: Determine whether the previous audio segment is valid data; If so, the noise floor energy of the previous sound segment is determined as the noise floor energy of the current sound segment; otherwise, the noise floor energy of the current sound segment is determined based on the noise floor energy of the previous sound segment and the average energy of the current sound segment.

3. The method according to claim 1, characterized in that, Also includes: Delete the audio segment preceding the previous audio segment from the audio data.

4. The method according to claim 1, characterized in that, The step of using a neural network model to identify sleep sounds from the valid data includes: The effective data is convolutionally processed using a one-dimensional convolutional layer of the neural network model to obtain convolutional features. The convolutional features are normalized using the normalization layer of the neural network model to obtain normalized features; The normalized features are downsampled using at least one sub-network module of the neural network model to obtain downsampled features. The sub-network module includes: a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separating convolutional layer. The downsampled features are pooled using the pooling layer of the neural network model to obtain pooled features; The pooled features are processed by the fully connected layer of the neural network model to obtain the sleep sound recognition result.

5. The method according to claim 1, characterized in that, Before using a neural network model to perform sleep sound recognition on the valid data, the method further includes: Acquire multiple sound sample data and multiple sound sample labels, wherein the sound sample labels are category labels of the sound sample data; Using the multiple sound sample data as training data and the multiple sound sample labels as training labels, the neural network is trained to obtain the neural network model; The step of acquiring multiple sound sample data and multiple sound sample tags includes: for each sound sample data in the multiple sound sample data, determining whether the sound sample data is valid based on the average energy; if so, labeling the sound sample data to obtain the corresponding sound sample tag.

6. The method according to claim 1, characterized in that, The valid data input to the neural network model is one-dimensional data.

7. The method according to any one of claims 1-6, characterized in that, The sleep sound recognition result includes: multiple category probabilities of the current sound segment; after obtaining the sleep sound recognition result, it also includes: Determine whether the difference between the maximum and minimum probabilities of multiple categories in the current sound segment is less than a preset threshold; If so, the current sound segment is uploaded to the server so that the server stores the current sound segment as a training dataset, or the current sound segment is used for sleep sound recognition to obtain the sleep sound recognition result of the current sound segment.

8. The method according to any one of claims 1-6, characterized in that, After obtaining the sleep sound recognition result, the method further includes: A sleep quality report may be generated based on the sleep sound recognition results, or a health assessment report may be generated based on the sleep sound recognition results.

9. A sleep sound recognition device, characterized in that, include: The valid data determination module is used to acquire sound data; Obtain the current sound segment from the sound data, and obtain the noise floor energy of the previous sound segment; The current sound segment is divided into multiple sound frames, and the average energy of each sound frame in the multiple sound frames is calculated; Determine whether the sum of the first effective quantity and the second effective quantity among the plurality of sound frames is greater than a preset quantity threshold. The first effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the average energy of the current sound segment is greater than a first ratio threshold. The second effective quantity is the number of sound frames whose ratio between the average energy of the sound frame and the noise floor energy of the previous sound segment is greater than a second ratio threshold. If so, the current sound segment is determined as valid data. The recognition result acquisition module is used to perform sleep sound recognition on the effective data using a neural network model to obtain sleep sound recognition results. The neural network model includes a one-dimensional pointwise convolutional layer and a one-dimensional depthwise separating convolutional layer.

10. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in any one of claims 1 to 8.

11. The electronic device according to claim 10, characterized in that, The electronic devices include: mobile devices, wearable devices, home appliances, or medical devices.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 8.

Images