Voice-based cough and / or user information detection method and device, computer readable storage medium, computer program product and terminal equipment

Abstract

The invention provides a voice-based cough and / or user information detection method and device, a computer readable storage medium, a computer program product and terminal equipment. The voice-based cough and / or user information detection method comprises the following steps: acquiring first voice data; the first voice data is preprocessed to obtain a converted first image, and the pixel value of each pixel in the first image corresponds to the intensity of each frequency on the Mel spectrum of the first voice data; and inputting the first image to a classification model to obtain a classification result, the classification result comprising a first part and / or a second part, the first part comprising whether the first voice data is cough sound, the second part comprising user information, and the user information comprising at least one of the following categories: gender and age. The invention provides a scheme for accurately recognizing the cough sound in the voice data and / or extracting effective user information from the voice data.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech-based cough and / or user information detection method and device, a computer-readable storage medium, a computer program product, and a terminal device. Background Art

[0002] In the prior art, the voiceprint in the user's voice can be used to identify the specific user's identity. This is because the voiceprint is a sound wave spectrum displayed by an electroacoustic instrument. It contains speech information and has the characteristics of stability, measurability and uniqueness. Everyone's vocal organs are different in size and shape, which leads to differences in voiceprints. Voiceprint recognition is to determine whether the speaker of two voices is the same person by comparing the voice of the speaker on the same phoneme, thereby realizing the function of "recognizing a person by hearing the voice".

[0003] However, how to recognize coughs and / or extract effective user attribute information from the user's voice is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a solution for accurately identifying cough sounds in voice data and / or extracting effective user information from voice data.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, a method for detecting a cough and / or user information based on speech is provided, and the method for detecting a cough and / or user information based on speech includes: acquiring first speech data; preprocessing the first speech data to obtain a converted first image, wherein a pixel value of each pixel in the first image corresponds to an intensity of each frequency on a Mel spectrum of the first speech data; inputting the first image into a classification model to obtain a classification result, wherein the classification result includes a first part and / or a second part, wherein the first part includes whether the first speech data is a cough sound, and the second part includes user information, wherein the user information includes at least one of the following categories: gender and age.

[0007] Optionally, when the first voice data is a cough sound, the second part also includes a cough type.

[0008] Optionally, preprocessing the first speech data includes: converting the first speech data into a Mel spectrum; normalizing the Mel spectrum; and mapping the normalized Mel spectrum to each color channel to obtain the first image.

[0009] Optionally, the classification model includes a backbone network, a feature fusion network and a detection head, the backbone network is used to extract features of the first image to obtain a feature map, the feature fusion network is used to fuse features of the feature map, the detection head is used to detect the fused features, determine a target detection frame, and output the classification result based on the first speech data in the target detection frame.

[0010] Optionally, the detection head includes a main classifier and multiple sub-classifiers, the main classifier is used to detect whether the first voice data in the target detection frame is a cough sound, and the multiple sub-classifiers respectively determine a category in the second part.

[0011] Optionally, the original result output by the classification model includes the position of at least one target detection frame, a first confidence of each target detection frame, at least one category label, and a second confidence of each category label; the speech-based cough and / or user information detection method also includes: screening the at least one target detection frame according to the first confidence to obtain at least one first detection frame; eliminating redundant first detection frames according to the intersection and union ratio between the at least one first detection frame to obtain at least one second detection frame; mapping the at least one second detection frame to a corresponding position in the first speech data to determine the position of the cough sound in the first speech data.

[0012] Optionally, the classification model is pre-trained using training data, where the training data includes second speech data, positions of cough sounds in the second speech data, and pre-marked category labels.

[0013] Optionally, the position of the cough sound in the second voice data is represented by information of a detection frame, and the information of the detection frame includes the coordinates of a center point, a length, and a height of the detection frame in the Mel spectrum of the second voice data.

[0014] In a second aspect, the present application also provides a voice-based cough and / or user information detection device, the voice-based cough and / or user information detection device comprising: an acquisition module, used to acquire first voice data; a preprocessing module, used to preprocess the first voice data to obtain a converted first image, the pixel value of each pixel in the first image corresponds to the intensity of each frequency on the Mel spectrum of the first voice data; a classification module, used to input the first image into a classification model to obtain a classification result, the classification result comprising a first part and / or a second part, the first part comprising whether the first voice data is a cough sound, the second part comprising user information, the user information comprising at least one of the following categories: gender, age.

[0015] Optionally, when the first voice data is a cough sound, the second part also includes a cough type.

[0016] Optionally, the preprocessing module also includes: a conversion unit, used to convert the first speech data into a Mel spectrum; a normalization processing unit, used to normalize the Mel spectrum; and a mapping unit, used to map the normalized Mel spectrum to each color channel to obtain the first image.

[0017] Optionally, the classification model includes a backbone network, a feature fusion network and a detection head, the backbone network is used to extract features of the first image to obtain a feature map, the feature fusion network is used to fuse features of the feature map, the detection head is used to detect the fused features, determine a target detection frame, and output the classification result based on the first speech data in the target detection frame.

[0018] Optionally, the detection head includes a main classifier and multiple sub-classifiers, the main classifier is used to detect whether the first voice data in the target detection frame is a cough sound, and the multiple sub-classifiers respectively determine a category in the second part.

[0019] Optionally, the original result output by the classification model includes the position of at least one target detection box, a first confidence of each target detection box, at least one category label, and a second confidence of each category label;

[0020] The speech-based cough and / or user information detection device further includes: a screening module, configured to screen the at least one target detection frame according to the first confidence level to obtain at least one first detection frame; a removal module, configured to remove redundant first detection frames according to an intersection-and-union ratio between the at least one first detection frame to obtain at least one second detection frame; and a mapping module, configured to map the at least one second detection frame to a corresponding position in the first speech data to determine a position of a cough sound in the first speech data.

[0021] Optionally, the classification model is pre-trained using training data, where the training data includes second speech data, positions of cough sounds in the second speech data, and pre-marked category labels.

[0022] Optionally, the position of the cough sound in the second voice data is represented by information of a detection frame, and the information of the detection frame includes the coordinates of a center point, a length, and a height of the detection frame in the Mel spectrum of the second voice data.

[0023] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. The computer program is executed by a processor to execute any one of the methods provided in the first aspect.

[0024] In a fourth aspect, a terminal device is provided, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to execute any one of the methods provided in the first aspect.

[0025] According to a fifth aspect, a computer program product is provided, on which a computer program is stored, and the computer program is executed by a processor to execute a method provided in the first aspect.

[0026] In a sixth aspect, an embodiment of the present application further provides a chip (or a data transmission device) on which a computer program is stored, and when the computer program is executed by the chip, the steps of the above method are implemented.

[0027] In the seventh aspect, an embodiment of the present application also provides a system chip, which is applied to a terminal, and the chip system includes at least one processor and an interface circuit, the interface circuit and the at least one processor are interconnected through lines, and the at least one processor is used to execute instructions to execute a method provided in the first aspect.

[0028] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0029] In the technical solution of the present application, by using the classification model to identify and classify the first voice data, on the one hand, it is possible to identify whether the first voice data is a cough sound and output it as the first part of the classification result, which can assist professionals in diagnosis; on the other hand, it is possible to determine the category of the first voice data, such as the gender and age of the source user of the first voice data, so as to obtain richer information from the first voice data, and further provide assistance for subsequent diagnosis and treatment. In addition, the technical solution of the present application realizes accurate recognition of information in voice data by converting voice into images for the classification model to recognize the images.

[0030] Furthermore, the technical solution of the present application designs the detection head of the classification model to enable it to support hierarchical classification tasks, first determine the target detection frame, and then output the classification result based on the first voice data in the target detection frame, thereby reducing computing resource overhead and improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a method for detecting cough and / or user information based on voice provided in an embodiment of the present application;

[0032] Figure 2 It is a structural diagram of a classification model provided in an embodiment of the present application;

[0033] Figure 3is a structural diagram of another classification model provided in an embodiment of the present application;

[0034] Figure 4 is a flow chart of another voice-based cough and / or user information detection method provided in an embodiment of the present application;

[0035] Figure 5 4 is a schematic diagram of the structure of a voice-based cough and / or user information detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] As described in the background art, how to recognize coughs from a user's voice and / or extract effective user attribute information is a technical problem that needs to be solved urgently.

[0037] In addition, cough is an evaluation indicator of the treatment effect or disease progression of certain diseases (such as asthma). Detailed and accurate cough status information (such as the number of coughs per hour, coughing time, etc.) has important clinical guidance significance for disease diagnosis. Studies have shown that intelligent cough monitoring equipment is more accurate than manual cough identification. The current intelligent cough monitoring equipment is mainly used for medical monitoring purposes, and patients need to wear complex equipment for monitoring, which undoubtedly brings inconvenience to users.

[0038] However, cough sounds contain little information, and how to extract effective user information from cough sounds is a technical problem that needs to be solved urgently.

[0039] In the technical solution of the present application, by using a classification model to identify and classify the first voice data, on the one hand, it is possible to identify whether the first voice data is a cough sound and output it as the first part of the classification result, which can assist professionals in diagnosis; on the other hand, it is possible to determine the category of the first voice data, such as the gender, age, or cough type of the source user of the first voice data, so as to obtain richer information from the first voice data and further provide assistance for subsequent diagnosis and treatment.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0041] See also Figure 1 , Figure 1 The method shown can be implemented or executed by a terminal device. The terminal device can be various devices with data processing capabilities. For example, the terminal device can be a mobile phone, a computer, a tablet computer, a portable device, a wearable device, a voice recorder, etc. For another example, the terminal device can be a sensor with an integrated processor, that is, the terminal device can be a sensor with data processing capabilities.

[0042] It should be noted that the above-mentioned terminal device is only an exemplary description, and the terminal device in this embodiment can be various forms of devices that will appear in the future, as long as it has a data processing function.

[0043] It should also be noted that the terminal device may also include a memory, and the processor may execute each method step by reading a computer program from the memory and executing each step described below when executing the computer program.

[0044] The method provided in this application specifically includes steps 101 to 103.

[0045] Step 101: Acquire first voice data.

[0046] In a specific implementation, the terminal device may receive a collection instruction, and the collection instruction is used to instruct the terminal device to collect the user's voice. In response to the received collection instruction, the sensor records the user's voice. Exemplarily, the collection instruction may be issued by the user whose voice is collected. That is, the terminal device may receive a collection instruction issued by the user, and in response to the collection instruction issued by the user, the terminal device records the user's voice, thereby obtaining audio data.

[0047] As another example, the collection instruction may also be issued by a third-party user, that is, the person who issues the collection instruction and the person whose voice is recorded may not be the same person.

[0048] Specifically, the original audio data collected by the terminal device may be a pulse code modulation (PCM) data stream. Before sending the audio data, the terminal device may first sample the PCM data stream to obtain the audio data. The sampling rate of the PCM data stream may be set according to actual application requirements, for example, the sampling rate may be 16000 Hz.

[0049] In a specific implementation, any feasible voice collection device, such as a microphone, can be used to collect the first voice data, without the user having to wear complicated equipment, and the operation is simple and convenient.

[0050] It should be noted that in the embodiments of the present application, the collected data are all anonymized data, and it is not possible to know which person's data is based on the collected data. In other words, when collecting audio data, no information related to the user's real identity will be obtained.

[0051] Step 102: preprocess the first speech data to obtain a converted first image. The pixel value of each pixel in the first image corresponds to the intensity of each frequency on the Mel spectrum of the first speech data, and the position of each pixel in the first image corresponds to the time-frequency in the Mel spectrum. Specifically, the higher the intensity of the frequency, the larger the pixel value of the pixel corresponding to the frequency.

[0052] The embodiment of the present application achieves accurate recognition of information in voice data by converting voice into images for the classification model to recognize the images.

[0053] In a specific implementation, the first speech data is first converted into a Mel spectrum. Specifically, the first speech data is divided into N frames according to a window of a second duration, such as 0.032s, at intervals of a first duration, such as 0.0016 seconds (s), where N is a positive integer. Each frame is first bitwise multiplied with a Hanning window of a second duration, such as 0.032s, and then a short-time Fourier transform is performed to obtain the spectrum of the first speech data. The power spectrum is obtained by taking the square of the absolute value of the spectrum. The power spectrum is converted to the Mel scale using a Mel filter bank. Thereby, the power spectrum of the linear frequency is redistributed to the Mel frequency, so that the low-frequency part has a higher resolution and the high-frequency part has a lower resolution.

[0054] Secondly, the Mel spectrum is normalized so that all frequency components are within a comparable range, while reducing the impact of high-frequency or low-frequency noise on the spectrum, and also helping to ensure that the color encoding in the first image can accurately reflect the distribution of the original data. The normalized Mel spectrum is a grayscale image.

[0055] Finally, the normalized Mel spectrum is mapped to each color channel to obtain the first image. This can be achieved by defining a mapping function between the value of the color channel and the frequency value in the Mel spectrum, which converts the frequency value (or Mel frequency value) into a specific color. For example, different colors can be used to represent different frequency ranges to create a color image (ie, RGB image). Each frequency component can be represented by a specific color, so that it can be intuitively displayed in the image.

[0056] For example, first map the grayscale value to the HSV color space. Specifically, the grayscale value is mapped to the H channel, and the values ​​of the S channel and the V channel are set to 1 (maximum value) to obtain more vivid colors. Then convert the HSV image back to the RGB color space. The process of converting HSV to RGB is roughly divided into the following steps:

[0057] 1. Calculate the parameters C, X, and m.

[0058] C=V×S,

[0059] X=C×(1-|(H / 60°)mod 2-1|),

[0060] m=VC.

[0061] Among them, C (Chroma) represents the intensity of color, X represents the intermediate variable whose value depends on the specific position of the hue, and m represents the lowest brightness value.

[0062] 2. Determine the initial values ​​of R, G, and B.

[0063] According to the value of the H channel, assign initial values ​​to R, G, and B.

[0064] If 0°≤H<60°: R'=C, G'=X, B'=0

[0065] If 60°≤H<120°: R'=X, G'=C, B'=0

[0066] If 120°≤H<180°: R'=0, G'=C, B'=X

[0067] If 180°≤H<240°: R'=0, G'=X, B'=C

[0068] If 240°≤H<300°: R'=X, G'=0, B'=C

[0069] If 300°≤H<360°: R'=C, G'=0, B'=X

[0070] 3. Calculate the final R, G, B values.

[0071] The final RGB value is calculated by adding m (i.e. the brightness offset):

[0072] R=(R'+m)×255,

[0073] G = (G' + m) × 255,

[0074] B=(B'+m)×255.

[0075] Through the above process, complex Mel spectrum data can be converted into easy-to-understand RGB images, which is helpful for the analysis and understanding of audio signals.

[0076] Step 103: Input the first image into the classification model to obtain a classification result.

[0077] In the embodiment of the present application, the classification result includes the first part and / or the second part, the first part includes whether the first voice data is a cough sound, and the second part includes user information, and the user information includes at least one of the following categories: gender, age. That is, the present application can detect whether the first voice data is a cough sound, and can also detect the user's information.

[0078] Furthermore, the user voice may be a cough sound or a normal speaking sound, etc. When the first voice data is a cough sound, the second part may further include a cough type, that is, the second part includes user information, and the user information includes at least one of the following categories: gender, age, and cough type.

[0079] Specifically, gender may include male and female, or any other appropriate gender type.

[0080] Specifically, age can include multiple age groups, such as 0-6 years old, 6-16 years old, 16-30 years old, 30-40 years old, etc.; age can also include multiple stages, such as children, teenagers, youth, middle-aged, elderly, etc.

[0081] Specifically, cough types may include dry cough and wet cough.

[0082] For example, the classification result may be that the first voice data is a cough sound, the gender is female, the age is 20-30, and the cough type is a dry cough.

[0083] For example, the classification result may be that the first voice data is not a cough sound.

[0084] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0085] It is understandable that, in a specific implementation, the voice-based cough and / or user information detection method can be implemented in the form of a software program, which runs in a processor integrated inside a chip or chip module. The method can also be implemented in the form of software combined with hardware, which is not limited in this application.

[0086] In this embodiment, by using the classification model to identify and classify the first voice data, on the one hand, it is possible to identify whether the first voice data is a cough sound and output it as the first part of the classification result, which can assist professionals in diagnosis; on the other hand, it is possible to determine the category of the first voice data, such as the gender, age, or cough type of the source user of the first voice data, so as to obtain richer information from the first voice data and further provide assistance for subsequent diagnosis and treatment.

[0087] It should be noted that the name of "classification model" does not limit the models mentioned in the embodiments of this application. In other embodiments, the above-mentioned "recognition model" may also be respectively referred to as "first model", "second model", etc., and this application does not impose any restrictions on this.

[0088] In this embodiment, a classification model may be deployed on the processor of the terminal device, and the classification model may be obtained by training based on a large amount of training data. The training data may include the second voice data and a pre-marked category label. Alternatively, the training data may also include the second voice data, the position of the cough sound in the second voice data, and a pre-marked category label.

[0089] For example, the training data includes the second speech data and pre-labeled category labels male and female.

[0090] For example, the training data includes the second voice data, and pre-labeled category labels 0-6 years old, 6-16 years old, 16-30 years old, 30-40 years old, etc.

[0091] For example, the training data includes the second speech data, the position of the cough sound, and pre-labeled category labels male and female, pre-labeled category labels 0-6 years old, 6-16 years old, 16-30 years old, 30-40 years old, etc., and pre-labeled category labels dry cough and wet cough.

[0092] Furthermore, the position of the cough sound in the second speech data is represented by information of a bounding box, and the information of the bounding box includes the coordinates of the center point, length, and height of the bounding box in the Mel spectrum of the second speech data. The length of the bounding box is the distance between the starting coordinates of the cough sound in the Mel spectrum and the ending coordinates of the cough sound in the Mel spectrum, and the height of the bounding box is the width of the Mel spectrum.

[0093] In this embodiment, in the process of training the classification model using training data, the preprocessing process of the second voice data is similar to the preprocessing process of the first voice data, that is, the second voice data is preprocessed to obtain a converted second image, and the second image will be input into the classification model to train the classification model.

[0094] The preprocessing process of the second voice data may refer to the aforementioned embodiment and will not be described in detail here.

[0095] Refer to Figure 2 , Figure 2 The structure of a classification model is shown, wherein the classification model 20 includes a backbone network 201 , a feature fusion network 202 and a detection head 203 .

[0096] The backbone network 201 is used to extract features of the input image. Specifically, the backbone network 201 extracts multi-scale feature maps from the input image, and these feature maps will be used for subsequent feature fusion and detection. Specifically, the input image can be the first image, or it can be the second image converted from the second speech data in the training phase.

[0097] The feature fusion network 202 is used to fuse the features of the feature graph. Specifically, the feature fusion network 202 can effectively fuse feature graphs from different scales to enhance the multi-scale detection capability of the model. The feature fusion network 202 can be a path aggregation network (PANet).

[0098] The detection head 203 is used to detect the fused features, determine the target detection frame, and output the classification result according to the first speech data in the target detection frame. The detection head 203 adopts a separate regression and classification design, which is responsible for the regression of the detection frame and the target classification respectively, and makes predictions based on the results of the feature fusion layer.

[0099] Specifically, this embodiment does not limit the feature extraction method.

[0100] Furthermore, the classification result includes a first part and a second part, the first voice data is a cough sound, and the second part also includes a cough type. Figure 3 , the detection head 203 includes a main classifier 2031 and multiple sub-classifiers 2032. The main classifier 2031 is used to detect whether the first voice data in the target detection frame is a cough sound, and the multiple sub-classifiers 2031 respectively determine a category in the second part. Among them, the main classifier 2031 is responsible for distinguishing whether the audio in the detection frame is a cough sound, and the sub-classifier 2032 is used when a cough sound is detected to further distinguish the gender, age group, dry cough or wet cough of the cough sound.

[0101] For example, the detection head 203 includes three sub-classifiers 2032. The three sub-classifiers 2032 are used to determine gender, age and cough type respectively.

[0102] For example, the detection head 203 includes two sub-classifiers 2032. The two sub-classifiers 2032 are used to determine age and cough type, respectively.

[0103] For example, the detection head 203 includes two sub-classifiers 2032. The two sub-classifiers 2032 are used to determine gender and cough type respectively.

[0104] It should be noted that the specific classifier 2032 can be set according to the actual application scenario, and this application does not impose any restrictions on this.

[0105] In a specific embodiment, the classification model 20 may be a You Only Look Once (YOLOX) model.

[0106] Accordingly, the function of the detection head 203 in the classification model will be different depending on the classification result.

[0107] For example, when the classification result includes the second part, for example, the classification result includes gender, the classification model detection head 203 may output a gender classification result according to the first speech data in the target detection frame.

[0108] For example, when the classification result includes the second part, for example, the classification result includes age, the classification model detection head 203 can output the age group classification result according to the first speech data in the target detection frame.

[0109] Continue to refer to Figure 3 , the detection head 203 may also include a separation regression head 204 and a confidence module 205. The separation regression head 204 is used for the regression of the target detection frame, that is, to determine the position of the target detection frame according to the output of the feature fusion network 202. The target detection frame may be represented by the coordinates of the center point of the target detection frame in the Mel spectrum of the first speech data, the length of the target detection frame, and the height of the target detection frame. The separation regression head 204 may output the position of at least one target detection frame.

[0110] The confidence module 205 is used to calculate the first confidence of each target detection frame and the second confidence of each category label. The confidence module 205 can output the first confidence of each target detection frame and the second confidence of each category label.

[0111] In a specific embodiment, the classification model 20 can be a YOLOX model or a residual network (Residual Network, ResNet) model, such as a ResNet-18 model. ResNet-18 is a deep learning network architecture based on a convolutional neural network that introduces residual learning to prevent network degradation, which can further ensure the accuracy of classification.

[0112] It should be noted that the classification model may also be a model constructed by any other feasible algorithm, and this application does not impose any limitation on this.

[0113] In a non-limiting embodiment, the raw results output by the classification model include the location of at least one target detection box, a first confidence of each target detection box, at least one category label, and a second confidence of each category label.

[0114] Taking the classification model YOLOX as an example, the feature fusion network 202 will generate three feature layers (e.g., P3, P4, P5), each with a different grid size (N1×N1, N2×N2, N3×N3). Each grid unit can generate n prediction results. Then the model's preliminary prediction results will be n×(N1×N1+N2×N2+N3×N3). Therefore, it is necessary to retain the streamlined classification results containing high-confidence targets in the above original results.

[0115] In a non-limiting example, see Figure 4 , Figure 4 A schematic diagram is shown for filtering out the final classification results from the original results.

[0116] Step 401: Filter at least one target detection frame according to a first confidence level to obtain at least one first detection frame.

[0117] Specifically, target detection frames whose first confidence is less than a preset threshold, such as 0.5 or 0.7, are preliminarily excluded, that is, target detection frames whose first confidence is greater than or equal to the preset threshold are retained to obtain at least one first detection frame.

[0118] Step 402: Eliminate redundant first detection frames according to an intersection-over-union ratio between at least one first detection frame to obtain at least one second detection frame.

[0119] In the specific implementation, the non-maximum suppression (NMS) algorithm removes redundant prediction boxes.

[0120] For example, we first exclude detection frames with confidence lower than 0.5, and then use non-maximum suppression to remove redundant detection frames to ensure that each cough sound is detected only once.

[0121] Specifically, the first detection frames are sorted from large to small according to the first confidence, and the first detection frame with the highest first confidence is selected as the reserved frame, which is considered to be the area in the current image that is most likely to contain the target. Then the intersection over union (IoU, defined as the ratio of the area of ​​the intersection and the area of ​​the union of two frames) of all remaining first detection frames and the current reserved frame is calculated. The first detection frame whose IoU with the current reserved frame exceeds a preset threshold, such as 0.5, is deleted. The first detection frame is considered to be a redundant detection, which usually indicates multiple predictions of the same target. Repeat the above steps from the remaining first detection frames until all first detection frames are processed. Finally, the retained first detection frame is output, that is, the second detection frame is obtained.

[0122] Step 403: Map at least one second detection frame to a corresponding position in the first speech data to determine the position of the cough sound in the first speech data.

[0123] In a specific implementation, the retained second detection frame is linearly mapped back to the position of the original audio (ie, the first voice data), and finally the position of each cough sound and the label of the cough sound are obtained.

[0124] Those skilled in the art will appreciate that steps 401 to 403 may be considered as the steps in the above Figure 1 The steps 101 to 103 described in the illustrated embodiment are executed afterward and are used to filter the original results output by the classification model.

[0125] In an optional embodiment, the steps 401 to 403 may also be performed by a classification model. Specifically, the classification model may include corresponding functional modules to respectively perform steps 401 to 403.

[0126] Please refer to Figure 5 , Figure 5 A speech-based cough and / or user information detection device 50 is shown. The speech-based cough and / or user information detection device 50 may include:

[0127] An acquisition module 501 is used to acquire first voice data;

[0128] A preprocessing module 502 is used to preprocess the first speech data to obtain a converted first image, wherein the pixel value of each pixel in the first image corresponds to the intensity of each frequency on the Mel spectrum of the first speech data;

[0129] The classification module 503 is used to input the first image into the classification model to obtain a classification result, where the classification result includes a first part and / or a second part, where the first part includes whether the first voice data is a cough sound, and the second part includes user information, where the user information includes at least one of the following categories: gender and age.

[0130] Further, the pre-processing module 502 may include:

[0131] A conversion unit, used for converting the first speech data into a Mel spectrum;

[0132] A normalization processing unit, used for normalizing the Mel spectrum;

[0133] A mapping unit is used to map the normalized Mel spectrum to each color channel to obtain a first image.

[0134] Furthermore, the voice-based cough and / or user information detection device 50 may further include:

[0135] A screening module, configured to screen at least one target detection frame according to a first confidence level to obtain at least one first detection frame;

[0136] A removal module, configured to remove redundant first detection frames according to an intersection-and-union ratio between at least one first detection frame, so as to obtain at least one second detection frame;

[0137] A mapping module is used to map at least one second detection frame to a corresponding position in the first voice data to determine the position of the cough sound in the first voice data.

[0138] Furthermore, the classification model is pre-trained using training data, where the training data includes the second speech data, the position of the cough sound in the second speech data, and a pre-marked category label.

[0139] Furthermore, the position of the cough sound in the second voice data is represented by information of a detection frame, and the information of the detection frame includes the coordinates of the center point, length, and height of the detection frame in the Mel spectrum of the second voice data.

[0140] Furthermore, the classification model includes a backbone network, a feature fusion network and a detection head. The backbone network is used to extract features of the first image to obtain a feature map. The feature fusion network is used to fuse features of the feature map. The detection head is used to detect the fused features, determine the target detection frame, and output the classification result based on the first speech data in the target detection frame.

[0141] Furthermore, the detection head includes a main classifier and a plurality of sub-classifiers, the main classifier is used to detect whether the first voice data in the target detection frame is a cough sound, and the plurality of sub-classifiers respectively determine a category in the second part.

[0142] In a specific implementation, the above-mentioned voice-based cough and / or user information detection device 50 may correspond to a chip with a voice-based cough and / or user information detection function in a terminal device, such as a system-on-a-chip (SOC), a baseband chip, etc.; or correspond to a chip module with a voice-based cough and / or user information detection function in a terminal device; or correspond to a chip module with a data processing function chip, or correspond to a terminal device.

[0143] For other related descriptions of the voice-based cough and / or user information detection device 50 , reference may be made to the related descriptions in the aforementioned embodiments, which will not be repeated here.

[0144] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component of the chip module (such as a chip, circuit module, etc.) or in different components, or at least some of the modules / units can be implemented in the form of software programs. It is implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in hardware such as circuits; for various devices and products applied to or integrated in the terminal equipment, the various modules / units contained therein can be implemented in hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal equipment, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal equipment, and the remaining (if any) modules / units can be implemented in hardware such as circuits.

[0145] The embodiment of the present application also discloses a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is run, the steps of the method shown in the above embodiment can be executed. The storage medium may include a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. The storage medium may also include a non-volatile memory (non-volatile) or a non-transitory memory, etc.

[0146] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0147] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0148] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0149] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0150] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means.

[0151] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0155] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present application.

[0156] Although the present application is disclosed as above, the present application is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application shall be subject to the scope defined by the claims.

Claims

1. A method for detecting cough and / or user information based on speech, characterized in that: include: Acquire first voice data; Preprocessing the first speech data to obtain a converted first image, wherein a pixel value of each pixel in the first image corresponds to an intensity of each frequency on a Mel spectrum of the first speech data; The first image is input into a classification model to obtain a classification result, wherein the classification result includes a first part and / or a second part, wherein the first part includes whether the first voice data is a cough sound, and the second part includes user information, wherein the user information includes at least one of the following categories: gender and age.

2. The method for detecting cough and / or user information based on speech according to claim 1, characterized in that: When the first voice data is a cough sound, the second part further includes a cough type.

3. The method for detecting cough and / or user information based on speech according to claim 1, characterized in that: The preprocessing of the first voice data comprises: Converting the first speech data into a Mel spectrum; Normalizing the Mel spectrum; The normalized Mel spectrum is mapped to each color channel to obtain the first image.

4. The method for detecting cough and / or user information based on speech according to claim 1, characterized in that: The classification model includes a backbone network, a feature fusion network and a detection head. The backbone network is used to extract features of the first image to obtain a feature map. The feature fusion network is used to fuse features of the feature map. The detection head is used to detect the fused features, determine a target detection frame, and output the classification result based on the first speech data in the target detection frame.

5. The method for detecting cough and / or user information based on speech according to claim 4, characterized in that: The detection head includes a main classifier and a plurality of sub-classifiers, wherein the main classifier is used to detect whether the first voice data in the target detection frame is a cough sound, and the plurality of sub-classifiers respectively determine a category in the second part.

6. The method for detecting cough and / or user information based on speech according to claim 4, characterized in that: The original result output by the classification model includes the position of at least one target detection frame, a first confidence of each target detection frame, at least one category label, and a second confidence of each category label; The voice-based cough and / or user information detection method further includes: Filtering the at least one target detection frame according to the first confidence level to obtain at least one first detection frame; Eliminate redundant first detection frames according to an intersection-and-union ratio between the at least one first detection frame to obtain at least one second detection frame; The at least one second detection frame is mapped to a corresponding position in the first voice data to determine the position of the cough sound in the first voice data.

7. The method for detecting cough and / or user information based on speech according to claim 1, characterized in that: The classification model is pre-trained using training data, where the training data includes second speech data, the position of cough sounds in the second speech data, and pre-marked category labels.

8. The method for detecting cough and / or user information based on speech according to claim 7, characterized in that: The position of the cough sound in the second voice data is represented by information of a detection frame, and the information of the detection frame includes the coordinates of a center point, a length, and a height of the detection frame in the Mel spectrum of the second voice data.

9. A voice-based cough and / or user information detection device, characterized in that: include: An acquisition module, used for acquiring first voice data; A preprocessing module, used for preprocessing the first speech data to obtain a converted first image, wherein the pixel value of each pixel in the first image corresponds to the intensity of each frequency on the Mel spectrum of the first speech data; A classification module is used to input the first image into a classification model to obtain a classification result, wherein the classification result includes a first part and / or a second part, wherein the first part includes whether the first voice data is a cough sound, and the second part includes user information, wherein the user information includes at least one of the following categories: gender and age.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the speech-based cough and / or user information detection method according to any one of claims 1 to 8 are performed.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the voice-based cough and / or user information detection method according to any one of claims 1 to 8 are implemented.

12. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the processor performs the steps of the speech-based cough and / or user information detection method according to any one of claims 1 to 8.

Images