Object identification method and device

Abstract

The embodiment of the invention provides an object recognition method and device, and the method comprises the steps: obtaining audio data, determining audio features corresponding to the audio data, and enabling the audio features to be used for indicating the features of the audio data in a time domain dimension and the features of the audio data in a frequency domain dimension. According to the audio features, the target type of the audio data is determined, and the target type is any one of the following preset types: speaking voice, breathing voice and silence. And determining a target model corresponding to the target type in a plurality of preset models, and inputting the audio features into the target model to obtain a judgment result output by the target model, the judgment result being used for indicating collected person information corresponding to the audio data. And determining whether the user using the terminal equipment is a juvenile user or not according to the judgment result. According to the technical scheme of the invention, the method can effectively achieve the recognition of the juvenile user based on the audio data.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an object recognition method and device. Background Art

[0002] With the continuous development of terminal equipment technology, the users of terminal equipment are becoming more and more extensive, and the users can include people of all ages.

[0003] Among the users of all age groups, the use of terminal devices by minors is particularly noteworthy, so it is particularly important to detect whether the user currently using the terminal device is a minor. Currently, there is no effective solution in the relevant technology to detect whether the user using the terminal device is a minor. Summary of the invention

[0004] The embodiments of the present application provide an object recognition method and device, which are applied in the field of terminal technology.

[0005] In a first aspect, an embodiment of the present application provides an object recognition method. The method includes:

[0006] Acquire audio data, and determine audio features corresponding to the audio data, where the audio features are used to indicate features of the audio data in a time domain dimension and a frequency domain dimension;

[0007] Determine the target type of the audio data according to the audio feature, wherein the target type is any one of the following preset types: speaking sound, breathing sound, and quiet sound;

[0008] Determine a target model corresponding to the target type from a plurality of preset models, and input the audio feature into the target model to obtain a determination result output by the target model, wherein the determination result is used to indicate the collected person information corresponding to the audio data;

[0009] According to the determination result, it is determined whether the user using the terminal device is a minor user.

[0010] By determining the target type of the audio data according to the audio features of the audio data, and then determining the corresponding target model according to the target type, the audio of breathing sounds and quiet sounds can be processed separately to improve the success rate of underage recognition. And then, according to the judgment result output by the target model, it is determined whether the user using the terminal device is a minor user, so that the identification of underage users based on audio data can be effectively realized.

[0011] In some implementations, determining the audio features corresponding to the audio data includes:

[0012] Performing short-time Fourier transform on the audio data to obtain a two-dimensional time-frequency signal corresponding to the audio data;

[0013] The two-dimensional time-frequency signal is determined as an audio feature corresponding to the audio data.

[0014] In this implementation, the two-dimensional time-frequency signal is directly determined as the audio feature, which can simply and quickly achieve the acquisition of the audio feature.

[0015] In some implementations, after performing a short-time Fourier transform on the audio data to obtain a two-dimensional time-frequency signal corresponding to the audio data, the method further includes:

[0016] Determining a time-frequency diagram according to the amplitude of the two-dimensional time-frequency signal;

[0017] Determine the Mel-frequency cepstrum corresponding to the audio data according to the time-frequency graph;

[0018] Performing discrete cosine transform on the Mel-cepstrum to obtain Mel-cepstrum coefficients corresponding to the audio data;

[0019] The Mel-cepstrum and the Mel-cepstrum coefficients are determined as audio features corresponding to the audio data.

[0020] In this implementation, the Mel-cepstrum and the Mel-cepstrum coefficients are determined as the audio features corresponding to the audio data, which can more finely express the low-frequency information of the audio data, so that the audio data of different age groups can be more effectively distinguished in the subsequent processing process.

[0021] In some implementations, determining the target type of the audio data according to the audio feature includes:

[0022] Dividing the Mel-frequency cepstrum into a plurality of segments according to a preset length and a preset step size;

[0023] For any of the segments, determining a probability that the segment corresponds to each of the preset types;

[0024] According to the probability that each of the segments corresponds to each of the preset types, a target type of the audio data is determined in each of the preset types.

[0025] In this implementation, the duration of each segment is relatively short, and the corresponding specific audio content is relatively simple, so it is easy to perform subsequent analysis and improve the accuracy of the probability that the determined segment corresponds to each preset type.

[0026] In some implementations, determining the probability that the segment corresponds to each of the preset types includes:

[0027] The segment is input into a recognition model to obtain the probability that the segment output by the recognition model corresponds to each of the preset types.

[0028] In some implementations, determining the target type of the audio data in each of the preset types according to the probability that each of the segments corresponds to each of the preset types includes:

[0029] Determine, according to the probability that each of the segments corresponds to each of the preset types, a probability mean corresponding to each of the preset types;

[0030] The target type of the audio data is determined according to the probability means corresponding to each of the preset types.

[0031] In some implementations, if the target type is speech, the target model is a speech recognition model, and the step of inputting the audio feature into the target model to obtain a determination result output by the target model includes:

[0032] The Mel-cepstral coefficients are input into the speech recognition model to obtain a determination result output by the speech recognition model, wherein the determination result includes at least one of the following: a first predicted age value, a first gender probability, a first minor probability, and a first feature vector.

[0033] In some implementations, if the target type is a breathing sound, the target model is a breathing sound recognition model, and the step of inputting the audio feature into the target model to obtain a determination result output by the target model includes:

[0034] The Mel-frequency cepstral coefficient is input into the respiratory sound recognition model to obtain a determination result output by the respiratory sound recognition model, wherein the determination result includes at least one of the following: a second predicted age value, a second gender probability, a second probability of being a minor, and a second eigenvector.

[0035] In this implementation, the breathing sound recognition model can achieve targeted processing of breathing sounds, and the speaking sound recognition model can achieve targeted processing of speaking sounds, by separately processing the audio of breathing sounds and the audio of quiet sounds to improve the success rate of identifying minors.

[0036] In some implementations, determining whether the user using the terminal device is a minor user according to the determination result includes:

[0037] Acquire supplementary data, the supplementary data including at least one of the following: image data collected during face recognition, inertial sensor data collected by an inertial sensor, touch data collected by a touch screen, application usage records, and fingerprint data collected during fingerprint authentication;

[0038] The determination result and the supplementary data are input into a decision model to obtain a recognition result output by the decision model, wherein the recognition result is used to indicate whether the user using the terminal device is a minor.

[0039] In this implementation, supplementary data is also integrated into the processing of the decision model, thereby effectively improving the accuracy of the final recognition result.

[0040] In a second aspect, an embodiment of the present application provides a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc.

[0041] The terminal device comprises: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method of the first aspect.

[0042] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0043] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the method of the first aspect.

[0044] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method described in the first aspect.

[0045] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of an object recognition method provided in an embodiment of the present application;

[0047] Figure 2 The process of the object recognition method provided in the embodiment of the present application Figure 2 ;

[0048] Figure 3 The process of the object recognition method provided in the embodiment of the present application Figure 3 ;

[0049] Figure 4 A schematic diagram of segment division provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of determining a probability mean provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a process for determining a recognition result provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the identity authentication interface provided in the embodiment of the present application;

[0053] Figure 8 A schematic diagram of an interface for canceling payment provided in an embodiment of the present application;

[0054] Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0055] Fig.10 This is a software structure block diagram of the terminal device according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0057] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0058] It should be noted that the "at..." in the embodiment of the present application can be the instant when a certain situation occurs, or a period of time after a certain situation occurs, and the embodiment of the present application does not specifically limit this. In addition, the display interface provided in the embodiment of the present application is only an example, and the display interface can also include more or less content.

[0059] With the continuous development of terminal device technology, the user population of terminal devices is becoming more and more extensive, including people of all ages, such as minors and adults, among which the adult population can be further divided into young people and old people.

[0060] Among the users of all ages, the use of terminal devices by minors is particularly noteworthy. For example, recharge functions are commonly found in various applications. If minors use the guardian's terminal device and the guardian's account to recharge in the application without the guardian's knowledge, controversial payment behavior will occur. Whether it is a recharge application that provides recharge services or a payment application that provides payment services, because the guardian's compliant account is used throughout the payment process, it is impossible to identify the user who actually made the payment as a minor through the account type.

[0061] For another example, there are some gaming applications that can set usage time and game content restrictions for minors. However, if the minor uses the guardian's terminal device and the guardian's account to play without the guardian's knowledge, the above restrictions can be avoided. Similarly, gaming applications cannot identify the user who actually operates the terminal device as a minor through the account type.

[0062] Therefore, it is particularly important to detect whether the user currently using the terminal device is a minor. There is no effective solution for detecting underage users during the use of terminal devices in the current related technologies.

[0063] In response to the problems in the prior art, the present application proposes the following technical concept: because there are certain differences between the voice characteristics of minors and adults, audio can be collected during the use of the terminal device, and then feature analysis can be performed based on the collected audio, so as to effectively identify whether the user currently using the terminal device is a minor.

[0064] Based on the above introduction, the object recognition method provided by the present application is described below in conjunction with specific embodiments. Figure 1 A flowchart of an object recognition method provided in an embodiment of the present application.

[0065] like Figure 1 As shown, the method includes:

[0066] S101. Acquire audio data, and determine audio features corresponding to the audio data, where the audio features are used to indicate features of the audio data in a time domain dimension and a frequency domain dimension.

[0067] In this embodiment, the terminal device can collect audio data during user use. In one possible implementation, the terminal device can collect audio data when it is determined that the collection conditions are met, where the collection conditions are, for example, detecting that the target application is started (mainly collecting audio of breathing sounds), or detecting that the user calls the microphone for voice input (mainly collecting audio of speaking sounds), etc. More specifically, it is possible to detect that the user calls the microphone for voice input in scenarios such as sound authentication, calling a voice assistant, making calls, game voice, social software voice, etc. This embodiment does not limit the specific scenarios for calling the microphone.

[0068] This embodiment does not limit the implementation method of the collection conditions, and the specific collection conditions can be selected and set according to actual needs.

[0069] Exemplarily, the audio acquisition parameters can be pre-set, and then the audio data is collected according to the pre-set audio acquisition parameters, wherein the audio acquisition parameters may include sampling rate, number of channels, sampling accuracy, sampling duration, etc. For example, the sampling rate can be set to 16kHz (kilohertz), the number of channels can be set to single channel, the sampling accuracy can be set to be normalized to the range of [-1,1], the accuracy is float (floating point number), and the sampling duration can be set to 2 to 5 seconds. In the actual implementation process, the specific audio sampling parameters and the specific settings of the audio sampling parameters can be selected according to actual needs, and this embodiment does not limit this.

[0070] After collecting the audio data, the terminal device can, for example, store the collected audio data for use when needed. And in a possible implementation, for example, a limit of no more than 10 times of audio collection per hour can be set to limit the amount of collected audio data to avoid collecting too much redundant data and causing system performance degradation.

[0071] In this embodiment, when certain trigger conditions are met, the terminal device can obtain audio data, where the trigger condition can, for example, be detecting a payment behavior initiated in the target application, or detecting a game behavior initiated in the target application. In other words, the trigger condition in this embodiment is a condition set for scenarios where underage detection is required, and the specific implementation method of the trigger condition can be selected and set according to actual needs.

[0072] When the audio data is obtained later, for example, the most recent N audio data may be obtained from the stored audio data, where N is an integer greater than or equal to 1. Alternatively, the audio data may be collected in real time at the current moment, which is not covered in this embodiment.

[0073] Secondly, feature extraction can be performed on the audio data to determine the audio features corresponding to the audio data. The audio features in this embodiment can indicate the features of the audio data in the time domain dimension and the frequency domain dimension.

[0074] It is understandable that there are certain differences in frequency between the voices of minors and adults, and this difference can be reflected by the characteristics of the frequency domain dimension, so the characteristics of the frequency domain dimension can effectively distinguish adults from minors. In addition, there are also certain differences in speech speed, intonation, etc. between the voices of minors and adults, and this difference can be reflected by the characteristics of the time domain dimension, so the characteristics of the time domain dimension can effectively distinguish adults from minors.

[0075] The audio features in this embodiment include both the features in the time domain dimension and the features in the frequency domain dimension, and thus can comprehensively reflect the features of the audio data.

[0076] S102. Determine a target type of the audio data according to the audio features, where the target type is any one of the following preset types: speaking sound, breathing sound, and quiet sound.

[0077] After determining the audio features of the audio data, in this embodiment, the target type of the audio data can be determined according to the audio features, wherein the target type is any one of the following preset types: speaking sound (speak), breathing sound (breath), and silence (silence).

[0078] That is to say, in this embodiment, it can be determined based on the audio features whether the collected audio data is of a speaking sound type, a breathing sound type, or a quiet sound type.

[0079] Among them, the speaking sound type refers to the audio mainly containing sounds with actual language content (speaking voices), the breathing sound type refers to the audio mainly containing breathing sounds, and the quiet sound type refers to the audio that has neither speaking voices nor breathing sounds for most of the time, which is mainly a quiet situation.

[0080] S103. Determine a target model corresponding to the target type from a plurality of preset models, and input the audio features into the target model to obtain a determination result output by the target model, wherein the determination result is used to indicate the collected person information corresponding to the audio data.

[0081] In this embodiment, a plurality of preset models are provided, wherein the preset models and the preset types have a corresponding relationship. Exemplarily, a corresponding preset model can be provided for each preset type, wherein the preset model is used to process the features of the audio data of the corresponding preset type, thereby determining the determination result of the audio data.

[0082] Therefore, after determining the target type of the audio data, the target model corresponding to the target type can be determined from multiple preset models. In the present embodiment, the preset models are all trained models, so they can be directly used to achieve the corresponding purposes. The preset models in the present embodiment are used to process the features of the corresponding type of audio data, thereby determining the judgment results of the audio data.

[0083] Afterwards, the audio features determined above can be input into the target model so that the target model outputs a judgment result, wherein the judgment result is used to indicate the collected person information corresponding to the audio data, and the collected person information refers to the relevant information of the user corresponding to the collected audio data. Exemplarily, the collected person information may include at least one of the following: predicted age value, gender probability, probability of being a minor, and feature vector.

[0084] S104: Determine whether the user of the terminal device is a minor based on the determination result.

[0085] The above-mentioned information of the person being collected can reflect whether the user is a minor in a certain dimension. Therefore, after determining the judgment result, it can be determined whether the user using the terminal device (that is, the user who triggers the trigger condition introduced above) is a minor user according to the judgment result, where the judgment result can specifically indicate whether the user is a minor user or not. The user using the terminal device introduced here is actually the person being collected corresponding to the audio data.

[0086] It should be noted that in different geographical regions, there may be some differences in the definitions of minors and adults. The minors and adults introduced in this embodiment are a relative concept, and the specific age boundaries of minors and adults can be selected and set according to actual needs. In other words, there is no strict age limit for minors and adults in this embodiment, and minors and adults can be defined according to actual needs.

[0087] The object recognition method provided by the embodiment of the present application includes: obtaining audio data, and determining the audio features corresponding to the audio data, the audio features are used to indicate the features of the audio data in the time domain dimension and the features of the frequency domain dimension. According to the audio features, the target type of the audio data is determined, and the target type is any one of the following preset types: speaking sound, breathing sound, and quiet sound. Determine the target model corresponding to the target type in multiple preset models, and input the audio features into the target model to obtain the determination result output by the target model, and the determination result is used to indicate the information of the person being collected corresponding to the audio data. According to the determination result, determine whether the user using the terminal device is a minor user. By determining the target type of the audio data according to the audio features of the audio data, and then determining the corresponding target model according to the target type, it is possible to process the audio of the breathing sound and the audio of the quiet sound separately to improve the success rate of the minor identification. And then determine whether the user using the terminal device is a minor user according to the determination result output by the target model, so that the identification of minor users based on audio data can be effectively realized.

[0088] Based on the above introduction, a possible implementation method for determining the audio features corresponding to the audio data is introduced below. Figure 2 To explain, Figure 2 The process of the object recognition method provided in the embodiment of the present application Figure 2 .

[0089] like Figure 2 As shown, the method includes:

[0090] S201. Perform short-time Fourier transform on audio data to obtain a two-dimensional time-frequency signal corresponding to the audio data.

[0091] In this embodiment, after the audio data is collected, the audio data may first be subjected to a short-time Fourier transform, wherein the short-time Fourier transform may convert the time domain signal into a two-dimensional time-frequency signal, where the two-dimensional time-frequency signal refers to a signal that includes both the time domain and the frequency domain. Therefore, in this embodiment, a two-dimensional time-frequency signal corresponding to the audio data may be obtained.

[0092] For example, when performing a short-time Fourier transform, the duration of the hanning window can be 25 milliseconds, and the step length of the window movement can be 12.5 milliseconds. In the actual implementation process, the specific duration of the hanning window and the specific setting of the step length can be selected according to actual needs, and this embodiment does not limit this.

[0093] In a possible implementation, the two-dimensional time-frequency signal can be directly determined as the audio feature corresponding to the audio data, wherein the two-dimensional time-frequency signal itself is a signal that includes both the time domain dimension and the frequency domain dimension, so the two-dimensional time-frequency signal can effectively indicate the features of the audio data in the time domain dimension and the frequency domain dimension. And it can be understood that directly determining the two-dimensional time-frequency signal as the audio feature can simply and quickly achieve the acquisition of the audio feature.

[0094] S202: Determine a time-frequency diagram according to the amplitude of the two-dimensional time-frequency signal.

[0095] Or further, in this embodiment, the time-frequency diagram may be further determined according to the amplitude of the two-dimensional time-frequency signal.

[0096] S203: Determine the Mel-frequency cepstrum corresponding to the audio data according to the time-frequency graph.

[0097] After determining the time-frequency graph, the frequency in the time-frequency graph is converted into the Mel scale to obtain the Mel spectrum corresponding to the audio data.

[0098] The mapping formula between frequency and Mel scale can refer to the following formula:

[0099]

[0100] Where f is the frequency and mel(f) is the Mel scale.

[0101] Exemplarily, in this embodiment, the minimum frequency is 0.0 Hz (Hertz), the maximum frequency is 8000.0 Hz, and the number of filter banks n_mels=64. Therefore, the Mel spectrum frequency domain after Mel scale conversion is divided into 64 scales, and the frequency range is [0.0, 8000] Hz.

[0102] Because people's perception of frequency is not linear, people perceive low-frequency sound signals better than high-frequency sound signals. Therefore, in this embodiment, the frequency is converted into the Mel scale, where the Mel scale changes exponentially. Therefore, the low-frequency information of the audio signal can be refined, and the high-frequency signal can be coarsened. Therefore, audio features that are easy to distinguish users of different ages can be extracted more effectively.

[0103] Furthermore, the mel-spectrum includes two parts, the envelope and the details. By taking the logarithm of the mel-spectrum determined above, the envelope and the details are separated, thereby obtaining a log mel-spectrum, also called a mel-cepstrum (log mel-specgram).

[0104] S204, performing discrete cosine transform on the Mel-cepstrum to obtain Mel-cepstrum coefficients corresponding to the audio data.

[0105] After obtaining the Mel-frequency cepstrum, the Mel-frequency cepstrum is further subjected to discrete cosine transform (DCT) to implement a simplified version of inverse discrete Fourier transform, thereby obtaining the Mel-frequency cepstrum coefficient (mfcc).

[0106] Exemplarily, in this embodiment, the first 40 coefficients after DCT can be retained to obtain Mel-cepstral coefficients. The Mel-cepstral obtained by the above audio data processing has a size of num_samples*n_mels, where num_samples is the number of samples of audio data and n_mels is the number of filter banks.

[0107] S205: Determine the Mel-cepstrum and the Mel-cepstrum coefficients as audio features corresponding to the audio data.

[0108] In a possible implementation, after the Mel-cepstrum and the Mel-cepstrum coefficients are determined, in this embodiment, the Mel-cepstrum and the Mel-cepstrum coefficients can be determined as audio features corresponding to the audio data.

[0109] Among them, the Mel-cepstral and Mel-cepstral coefficients can effectively reflect the characteristics of audio data in the time domain and frequency domain dimensions on the one hand, and on the other hand, they can more finely express the low-frequency information of audio data, so that audio data of different age groups can be more effectively distinguished in the subsequent processing process.

[0110] The above embodiment introduces the implementation method of determining the audio features of audio data. Figures 3 to 5 This section introduces how to determine the target type of audio data. Figure 3 The process of the object recognition method provided in the embodiment of the present application Figure 3 , Figure 4 A schematic diagram of segment division provided in an embodiment of the present application, Figure 5 A schematic diagram of determining a probability mean provided in an embodiment of the present application.

[0111] like Figure 3 As shown, the method includes:

[0112] S301 , dividing the Mel-frequency cepstrum into a plurality of segments according to a preset length and a preset step size.

[0113] In this embodiment, a preset length and a preset step size may be set, wherein the preset length may be understood as the duration of the moving window, and the moving window may be moved by the preset step size each time, and each movement may obtain a segment of the Mel-cepstral spectrum, wherein the duration of one segment is the preset length.

[0114] For example, you can refer to Figure 4 To understand, such as Figure 4 As shown, for example, a rectangular frame 401 is used to indicate a Mel-cepstrum, and a moving window with a preset length a can be used to divide the Mel-cepstrum into segments. For example, a moving window with a preset length a is used for the first division in the Mel-cepstrum to obtain a segment 1 with a duration a.

[0115] After that, move the moving window Figure 4 The preset step size b is shown, and then a moving window with a preset length a is used to divide the Mel-cepstrum, and a segment 2 with a duration a is obtained. Then, the moving window is continuously moved according to the preset step size until the segment division of the Mel-cepstrum is completed, thereby dividing the Mel-cepstrum into multiple segments.

[0116] It should also be noted that when the Mel-cepstrum cannot be divided into an integer number of segments, the Mel-cepstrum may be padded with zeros, and then the zero-padded Mel-cepstrum may be divided into an integer number of segments.

[0117] Exemplarily, the preset length can be set to 0.96 seconds, and the preset step size can be set to 0.24 seconds. In actual implementation, the values ​​of the preset length and the preset step size can be selected according to actual needs.

[0118] S302: For any segment, determine the probability that the segment corresponds to each preset type.

[0119] Based on the above introduction, it can be determined that the following preset types are set in this embodiment: speaking sound, breathing sound, and quiet sound. In this embodiment, for any segment, the probability of the segment corresponding to each preset type can be determined. And it can be understood that the segment in this embodiment is actually a segment cut out from the Mel cepstrum, which is essentially the audio feature data of the audio data. Therefore, the probability of the segment corresponding to each preset type introduced here actually expresses the probability that the audio data corresponding to the segment is of each preset type.

[0120] For example, in this embodiment, the probability that the type of audio data corresponding to the segment is speech sound, the probability that the type of audio data corresponding to the segment is breathing sound, and the probability that the type of audio data corresponding to the segment is silence sound are determined.

[0121] In a possible implementation, a recognition model may be pre-trained in this embodiment, the input of the recognition model is a segment obtained by Mel-frequency cepstrum, and the output is the probability that the segment corresponds to each preset type.

[0122] Therefore, for any segment, the segment can be input into the recognition model, and then the probability of the segment output by the recognition model corresponding to each preset type can be obtained. For example, for segment 1, the probability that segment 1 corresponds to the speech type (that is, the probability that the part of the audio data corresponding to segment 1 is the speech type), the probability that segment 1 corresponds to the breathing type (that is, the probability that the part of the audio data corresponding to segment 1 is the breathing type), and the probability that segment 1 corresponds to the quiet type (that is, the probability that the part of the audio data corresponding to segment 1 is the quiet type) can be obtained.

[0123] Exemplarily, the recognition model in this embodiment can be a Yamnet model, where Yamnet is a pre-trained deep network that uses deep separable convolution. Deep separable convolution is to super-divide ordinary convolution into a deep convolution and a point-by-point convolution. Compared with traditional convolution, it has a smaller number of parameters and is suitable for deployment as an end-side model. Using multiple deep separable convolution layers, sound features in multiple Mel-cepstral fragments can be extracted, and the sound features are converted into 1×1 feature data using a pooling layer, and then the probability of each preset type of each Mel-cepstral fragment is output through a fully connected layer and softmax (normalization function).

[0124] Exemplarily, the dimension of the Mel-cepstrum segment in this embodiment is, for example, 20×77×64. To facilitate subsequent processing, the segments can be concatenated in the order of the third dimension, the second dimension, and the first dimension, thereby converting the segments into a one-dimensional matrix with a dimension of 1×98560.

[0125] Then, the above one-dimensional matrix is ​​used as input to first obtain activation data, whose dimension is, for example, 1×10420.

[0126] In this embodiment, the activation obtained by processing the audio data can also be trimmed. For example, for 2 seconds of audio data, the first 2605 values, that is, the first 5 rows, can be retained; for another example, for 5 seconds of audio data, the first 9378 values, that is, the first 18 rows, can be retained.

[0127] The cropped activation is then converted into a two-dimensional matrix, where every 521 values ​​are 1 row. For example, for 2 seconds of audio data, a two-dimensional matrix with a dimension of 5×521 can be obtained; for 5 seconds of audio data, a two-dimensional matrix with a dimension of 18×521 can be obtained.

[0128] Then calculate the mean of each column in the two-dimensional matrix to get the average activation, whose dimension is 1 × 521. Then you can get the processing result output by the Yamnet model, where the 459th value is the probability that the segment corresponds to a quiet sound, the first value is the probability that the segment corresponds to a spoken sound, and the sum of the 37th to 46th values ​​is the probability that the segment corresponds to a breathing sound.

[0129] The dimensions and values ​​described above are only examples. In actual implementation, the dimensions and related values ​​can be adjusted according to actual needs. The processing flow of the recognition model can be executed according to the above description.

[0130] It is understandable that because the specific audio content contained in a longer audio data segment may be relatively complex, after the segment is divided, the duration of each segment can be made relatively short, and the composition of the corresponding specific audio content is also relatively simple, which is convenient for subsequent analysis and improves the accuracy of the probability that the currently determined segment corresponds to each preset type.

[0131] S303: Determine the probability mean corresponding to each preset type according to the probability that each segment corresponds to each preset type.

[0132] After determining the probability mean of the probabilities corresponding to the preset types for each segment respectively, the probability mean corresponding to each preset type can be determined according to the probability that each segment corresponds to the preset type.

[0133] For example, you can refer to Figure 5 To understand, such as Figure 5As shown, it is assumed that there are currently three segments, segment 1, segment 2 and segment 3, among which the probability that segment 1 corresponds to the speech type is 70%, the probability that segment 1 corresponds to the breathing type is 20%, and the probability that segment 1 corresponds to the quiet type is 10%. The probability that segment 2 corresponds to the speech type is 90%, the probability that segment 2 corresponds to the breathing type is 8%, and the probability that segment 2 corresponds to the quiet type is 2%. The probability that segment 3 corresponds to the speech type is 40%, the probability that segment 3 corresponds to the breathing type is 40%, and the probability that segment 3 corresponds to the quiet type is 20%.

[0134] Then, for each of the above probabilities, the probability mean is calculated for the preset type "speech voice", that is, the probability of segment 1 corresponding to speech voice, the probability of segment 2 corresponding to speech voice, and the probability of segment 3 corresponding to speech voice are calculated to obtain Figure 5 The average probability is 70%. And similarly for breath sounds and quiet sounds, Figure 5 In the case shown in FIG, the average probability corresponding to breath sounds is 22.7%, and the average probability corresponding to quiet sounds is 10.6%.

[0135] S304: Determine the target type of the audio data according to the probability mean values ​​corresponding to the preset types.

[0136] Afterwards, the target type of the audio data is determined according to the probability mean values ​​corresponding to each preset type. For example, the determination may be made according to the following rules:

[0137] If the mean probability corresponding to the silent sound is less than 0.5, and the mean probability corresponding to the spoken sound is greater than or equal to 0.15, then it is determined that the target type of the audio data is the spoken sound.

[0138] When the probability mean corresponding to the quiet sound is less than 0.5, the probability mean corresponding to the speaking sound is less than 0.1, and the probability mean corresponding to the breathing sound is greater than or equal to 0.04, it is determined that the target type of the audio data is the breathing sound.

[0139] If the two conditions described above are not met, the target type of the audio data is determined to be silence.

[0140] Because in actual scenarios, when users output sound, there are naturally certain differences in the proportion of speaking time, breathing time, and quiet time in an audio segment. Therefore, determining the target type of audio data according to the above rules can effectively ensure the rationality and accuracy of the determined target type.

[0141] For example, refer to Figure 5 For example, it can be determined that the target type of the audio data is spoken voice.

[0142] In this embodiment, the Mel cepstrum is divided into multiple segments, and then the probability corresponding to each preset type is determined for each segment, and then the average is calculated based on the probabilities corresponding to each segment to obtain the probability average corresponding to each preset type, and then the preset type with the largest probability average is determined as the target type of the audio data. Because the duration of the audio data corresponding to a single segment is relatively short, the accuracy of determining the probability corresponding to each preset type for a single segment is relatively high, and then the probability average is calculated based on each segment, and the preset type with the largest probability average is determined as the target type of the audio data, thereby effectively ensuring the accuracy of the determined target type.

[0143] Based on the above introduction, Figure 6 The implementation method of determining the judgment result according to the target model and the implementation method of finally determining the recognition result are explained. Figure 6 A schematic diagram of a process for determining a recognition result provided in an embodiment of the present application.

[0144] like Figure 6 As shown, in this embodiment, a speech recognition model is set for the preset type of speech sound, and a breathing sound recognition model is set for the preset type of breathing sound.

[0145] When the target type of the audio data is determined to be speech, the target model can be determined to be a speech recognition model, wherein the input of the speech recognition model is the Mel cepstral coefficient, and the output is the determination result corresponding to the audio data.

[0146] Then refer to Figure 6 In this embodiment, the Mel-cepstral coefficients determined above can be input into a speech recognition model to obtain a judgment result output by the speech recognition model. The judgment result output by the speech recognition model includes at least one of the following: a first predicted age value, a first gender probability, a first minor probability, and a first feature vector.

[0147] Among them, the first predicted age value is the age value predicted for the person collected of the audio data, the first gender probability is the probability that the person collected of the audio data is predicted to be male (or female), the first minor probability is the probability that the person collected of the audio data is predicted to be a minor, and the first eigenvector is the eigenvector of the audio data determined according to the Mel cepstral coefficient, which can be understood as a eigenvector focusing on the speaking voice.

[0148] Special explanation is required for the first predicted age value here. In a possible implementation method, in order to facilitate the model training process, during the training of the speech recognition model, the age data in the training set can be standardized, for example, by the following method: (real age - mean age of the training set) / standard deviation of age of the training set.

[0149] Therefore, the predicted age value directly output by the speech recognition model needs to be de-standardized, for example, by the following method: directly output predicted age value * training set age standard deviation + training set age mean, to obtain the first predicted age value introduced here.

[0150] For example, because people of different ages and genders have differences in speaking speed, voice, and intonation, a multi-scale convolutional model can be used to implement the currently introduced speech recognition model to extract speech features at different scales.

[0151] For example, the Mel cepstral coefficients can be used as model input, and multiple convolutional layers and attention layers of different scales can be used to extract speech features respectively. The fused speech features of different scales (which can be understood as serial concatenation) are averaged and pooled to output 1*384-dimensional features (that is, the first feature vector introduced above). The features are output through the fully connected layer and softmax to output the first predicted age value, first gender probability, and first probability of minors introduced above. Among them, convolution at different scales can obtain more comprehensive information on speech, thereby effectively improving the accuracy of the first predicted age value, first gender probability, and first probability of minors output in the end.

[0152] In a possible implementation, in this embodiment, the mfcc can be converted into a one-dimensional matrix, for example, starting from the first dimension, and then the converted matrix is ​​used as the input of the speech recognition model. Among them, for the mfcc of 2 seconds of audio data, the size of the converted one-dimensional matrix is, for example, 1×6440. For the mfcc of 5 seconds of audio data, the size of the converted one-dimensional matrix is, for example, 1×16040. For this, for example, only the 4800th to the 11239th can be retained, that is, only 6440 values ​​can be retained, so that audio data of various sizes can conform to the input format of the speech recognition model.

[0153] In the actual implementation process, the specific matrix size and the selection interval in the matrix can be selected and set according to actual needs.

[0154] Alternatively, when the target type of the audio data is determined to be breathing sound, the target model can be determined to be a breathing sound recognition model, wherein the input of the breathing sound recognition model is the Mel cepstral coefficient, and the output is the determination result corresponding to the audio data.

[0155] Also refer to Figure 6 In this embodiment, the above-determined Mel-cepstral coefficients can be input into the respiratory sound recognition model to obtain a determination result output by the respiratory sound recognition model. The determination result output by the respiratory sound recognition model includes at least one of the following: a second predicted age value, a second gender probability, a second minor probability, and a second eigenvector.

[0156] Among them, the second predicted age value is the age value predicted for the person collected by the audio data, the second gender probability is the probability that the person collected by the audio data is predicted to be male (or female), the second probability of minors is the probability that the person collected by the audio data is predicted to be a minor, and the second eigenvector is the eigenvector of the audio data determined according to the Mel cepstral coefficient, which can be understood as a eigenvector focusing on breathing sounds. The second predicted age value is similar to the above introduction and needs to be de-standardized to be obtained, which will not be repeated here.

[0157] For example, the differences in breathing sounds among people of different ages and genders mainly come from the degree of lung function development, vital capacity, and breathing habits. Therefore, the Mel-frequency cepstral coefficients can be used as model input, and frame-level features can be extracted using a multi-layer residual network. The attention statistical pooling layer is used to assign different weights to different frames, generate weighted averages and weighted standard deviations, and form 1*64-dimensional breathing detail features (that is, the second feature vector introduced above) through average pooling. The features are output through the fully connected layer and softmax to form the second predicted age value, second gender probability, and second probability of minors introduced above.

[0158] Similar to the above, in this embodiment, the mfcc can also be converted into a one-dimensional matrix, for example, starting from the first dimension and sequentially splicing, and then the converted matrix is ​​used as the input of the respiratory sound recognition model. Among them, for the mfcc of 5 seconds of audio data, the size of the converted one-dimensional matrix is, for example, 1×16040. In the actual implementation process, the specific matrix size can be selected and set according to actual needs.

[0159] It should be noted that, in the present embodiment, a breathing sound recognition model is pre-trained for breathing sounds, and a speaking sound recognition model is pre-trained for speaking sounds. Then, the breathing sound recognition model can realize targeted processing of breathing sounds, and the speaking sound recognition model can realize targeted processing of speaking sounds. That is to say, by realizing separate processing of the audio of breathing sounds and the audio of quiet sounds, the success rate of identifying minors can be effectively improved.

[0160] Whether it is the speech recognition model or the breathing sound recognition model introduced above, the predicted age value, gender probability and underage probability in the output discrimination results can reflect the probability that the user using the terminal device is a minor user to a certain extent. Therefore, for example, it is possible to directly determine whether the user using the terminal device is a minor user based on a single item of data in the discrimination result.

[0161] Alternatively, the various data in the discrimination result may be integrated to determine whether the user using the terminal device is a minor. Figure 6 The discrimination result can be input into the decision model, which makes a fusion decision and outputs a recognition result. The recognition result is used to indicate whether the user using the terminal device is a minor.

[0162] Alternatively, in order to further improve the accuracy of the final recognition result, additional data can be obtained, referring to Figure 6 , where the supplementary data includes at least one of the following: image data collected during face recognition, inertial sensor data collected by inertial sensors, touch data collected by touch screens, application (APP) usage records, and fingerprint data collected during fingerprint authentication. The supplementary data here can specifically be relevant data within a preset time period, or can also be the most recent M relevant data, where M is an integer greater than or equal to 1. This embodiment does not limit the specific collection method of the supplementary data.

[0163] It can be understood that the supplementary data introduced here can reflect the age characteristics of the user who is using the terminal device to a certain extent. Therefore, in the actual implementation process, the specific setting of the supplementary data can be selected according to actual needs. As long as the data can reflect the age characteristics, it can be used as supplementary data in this embodiment.

[0164] Then the judgment results introduced above and the supplementary data obtained here are input into the decision model to obtain the recognition results output by the decision model. Because the supplementary data introduced above are also integrated in the processing process of the decision model in the current case, the accuracy of the final recognition result can be effectively improved.

[0165] And in a possible implementation method, based on the supplementary data introduced above and the owner information stored in the terminal device itself (such as pre-stored facial information, historical inertial sensor data, historical touch data, historical APP usage records, pre-stored fingerprints), this embodiment can, for example, further identify whether the collected user corresponding to the supplementary data is the owner himself, thereby determining whether the user currently using the terminal device is the owner himself.

[0166] Based on the above-mentioned embodiments, possible application scenarios of the present application are exemplarily described below.

[0167] It is understandable that, usually, the terminal device will trigger the specific process of executing the object recognition method introduced in this application only when it determines that certain trigger conditions are met.

[0168] The trigger condition may be actively detected by the terminal device, for example, the terminal device detects a payment behavior initiated in the target application, or the terminal device detects a game behavior initiated in the target application, etc.

[0169] Alternatively, the trigger condition may also be detected by an application, for example, the target application detects the initiation of a payment behavior, or the target application detects the initiation of a game behavior, etc. In this case, the target application may send trigger information to the recognition unit in the application layer introduced above, so that the recognition unit starts executing the object recognition method in the present application.

[0170] That is to say, the trigger conditions in this embodiment are conditions set for scenarios where underage detection is required, and the specific implementation method of the trigger conditions can be selected and set according to actual needs.

[0171] After determining the recognition results introduced above, different processing methods can be used according to the application scenarios corresponding to different trigger conditions. Figure 7 and Figure 8 Several possible application scenarios are described. Figure 7 This is a schematic diagram of the identity authentication interface provided in the embodiment of the present application. Figure 8 A schematic diagram of the payment cancellation interface provided in an embodiment of the present application.

[0172] For example, a payment behavior is initiated in the target application and the corresponding triggering conditions are met, so that the identification unit executes the subsequent minor identification process, and then the terminal device can send the identification result of whether the person is a minor to the target application.

[0173] If the recognition result indicates that the user currently using the terminal device is an adult, the payment behavior can be performed normally; however, if the recognition result indicates that the user currently using the terminal device is a minor, the target application can intercept the payment behavior and pop up a reminder box on the payment interface, refer to Figure 7 , you can ask the user to complete further identity authentication, such as face recognition or fingerprint recognition and other payment authentication behaviors. If the user clicks Figure 7 , the identity authentication process is triggered. If the user clicks Figure 7 If you cancel the payment, the payment will be terminated.

[0174] After that, if the payment authentication behavior such as face recognition or fingerprint recognition is passed, you can continue to pay; and refer to Figure 8 If the authentication fails, the target application can cancel the payment and inform the user that the payment has been canceled, and can also send a reminder message to the mobile phone number corresponding to the login account in the current target application.

[0175] For another example, a gaming behavior is initiated in the target application and the corresponding triggering conditions are met, so that the identification unit performs subsequent identification of minors. The terminal device can then send the identification result of whether the person is a minor to the target application.

[0176] If the recognition result indicates that the user currently using the terminal device is an adult, the game behavior can be performed normally; however, if the recognition result indicates that the user currently using the terminal device is a minor, the target application can intercept the game behavior and pop up a reminder box on the game interface, requiring the user to complete payment authentication behaviors such as face recognition or fingerprint recognition. Afterwards, if the payment authentication behaviors such as face recognition or fingerprint recognition are authenticated, the game behavior can continue; if the authentication fails, the target application can cancel the game behavior and send a reminder message to the mobile phone number corresponding to the login account in the current target application.

[0177] Furthermore, the terminal device may also send the above-mentioned information about whether the user currently using the terminal device is the owner to the target application, and the target application integrates the above-mentioned identification result to determine whether to provide the corresponding service.

[0178] It should also be noted that the various embodiments in this application will only be executed if the user has turned on the underage identification function or underage identification permission, so as to effectively protect the user's data security.

[0179] Based on the above introduction, it can be determined that the technical solution of the present application can effectively identify whether the user currently using the terminal device is a minor user based on the collected audio data, so that when the minor user initiates some questionable behavior, it can be quickly and effectively intercepted and reminded to avoid the occurrence of controversial behaviors and protect the user's equipment and financial security.

[0180] An embodiment of the present application provides a terminal device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0181] Furthermore, the terminal device may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0182] The terminal device has the function of displaying an interface and the function of acquiring an image.

[0183] In order to better understand the embodiment of the present application, the structure of the terminal device in the embodiment of the present application is introduced below:

[0184] Fig. 9A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0185] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0186] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0187] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0188] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0189] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuits sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0190] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0191] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0192] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0193] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0194] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0195] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0196] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0197] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0198] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0199] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0200] Gyro sensor 180B is used to collect angle change data. Gyro sensor 180B is used to collect angle data.

[0201] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in all directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.

[0202] The acceleration change data is collected based on the acceleration sensor 180E. The terminal device determines the posture change information of the terminal device based on the acceleration change data collected by the acceleration sensor 180E.

[0203] The terminal device determines the screen direction based on the angle data collected by the gyro sensor 180B.

[0204] The terminal device determines the posture change information of the terminal device based on the angle change data collected by the gyroscope sensor 180B. Alternatively, the terminal device determines the posture change information of the terminal device based on the acceleration change data collected by the acceleration sensor 180E. Alternatively, the terminal device determines the posture change information of the terminal device based on the angle change data collected by the gyroscope sensor 180B and the acceleration change data collected by the acceleration sensor 180E.

[0205] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0206] The proximity light sensor 180G collects proximity light data. The terminal device determines proximity light information based on the proximity light data collected by the light sensor 180G.

[0207] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0208] The ambient light sensor 180L collects ambient light data. The terminal device determines ambient light information based on the ambient light data collected by the ambient light sensor 180L.

[0209] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0210] Fig.10 This is a software structure block diagram of the terminal device 100 according to an embodiment of the present application.

[0211] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0212] The application layer can include a series of application packages.

[0213] like Fig.10 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, wireless local area network (WLAN), Bluetooth, music, video, short message, lock screen application, setting application, etc. Of course, the application layer may also include other application packages, such as payment application, shopping application, banking application, game application, social application and other third-party applications, which are not limited in this application. Among them, the third-party application may have functions such as face recognition, video call, scanning, photo taking and video recording.

[0214] In this embodiment, the target application can be Fig.10 The payment application or game application shown in FIG.

[0215] And reference Fig.10 The application layer in this embodiment also includes an identification unit, where the identification unit is used to execute the minor identification process introduced in the above embodiments. The specific processing flow of the identification unit can refer to the introduction of the above embodiments and will not be repeated here.

[0216] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0217] like Fig.10 As shown, the application framework layer may include a window manager, a content provider, a resource manager, a view system, a notification manager, an activity manager, and the like.

[0218] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, capture the screen, etc.

[0219] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0220] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0221] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0222] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0223] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0224] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0225] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0226] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0227] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0228] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0229] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0230] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0231] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0232] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the protection scope of the present invention.

Claims

1. An object recognition method, characterized in that: include: Acquire audio data, and determine audio features corresponding to the audio data, where the audio features are used to indicate features of the audio data in a time domain dimension and a frequency domain dimension; Determine the target type of the audio data according to the audio feature, wherein the target type is any one of the following preset types: speaking sound, breathing sound, and quiet sound; Determine a target model corresponding to the target type from a plurality of preset models, and input the audio feature into the target model to obtain a determination result output by the target model, wherein the determination result is used to indicate the collected person information corresponding to the audio data; According to the determination result, it is determined whether the user using the terminal device is a minor user.

2. The method according to claim 1, characterized in that The determining the audio features corresponding to the audio data includes: Performing short-time Fourier transform on the audio data to obtain a two-dimensional time-frequency signal corresponding to the audio data; The two-dimensional time-frequency signal is determined as an audio feature corresponding to the audio data.

3. The method according to claim 2, characterized in that After performing short-time Fourier transform on the audio data to obtain a two-dimensional time-frequency signal corresponding to the audio data, the method further includes: Determining a time-frequency diagram according to the amplitude of the two-dimensional time-frequency signal; Determine the Mel-frequency cepstrum corresponding to the audio data according to the time-frequency graph; Performing discrete cosine transform on the Mel-cepstrum to obtain Mel-cepstrum coefficients corresponding to the audio data; The Mel-cepstrum and the Mel-cepstrum coefficients are determined as audio features corresponding to the audio data.

4. The method according to claim 3, characterized in that The step of determining the target type of the audio data according to the audio feature includes: Dividing the Mel-frequency cepstrum into a plurality of segments according to a preset length and a preset step size; For any of the segments, determining a probability that the segment corresponds to each of the preset types; According to the probability that each of the segments corresponds to each of the preset types, a target type of the audio data is determined in each of the preset types.

5. The method according to claim 4, characterized in that The determining the probability that the segment corresponds to each of the preset types includes: The segment is input into a recognition model to obtain the probability that the segment output by the recognition model corresponds to each of the preset types.

6. The method according to claim 4, characterized in that The determining the target type of the audio data in each of the preset types according to the probability that each of the segments corresponds to each of the preset types comprises: Determine, according to the probability that each of the segments corresponds to each of the preset types, a probability mean corresponding to each of the preset types; The target type of the audio data is determined according to the probability means corresponding to each of the preset types.

7. The method according to any one of claims 3 to 6, characterized in that: If the target type is speech, the target model is a speech recognition model, and the step of inputting the audio feature into the target model to obtain a determination result output by the target model includes: The Mel-cepstral coefficients are input into the speech recognition model to obtain a determination result output by the speech recognition model, wherein the determination result includes at least one of the following: a first predicted age value, a first gender probability, a first minor probability, and a first feature vector.

8. The method according to any one of claims 3 to 6, characterized in that: If the target type is a breathing sound, the target model is a breathing sound recognition model, and the step of inputting the audio feature into the target model to obtain a determination result output by the target model includes: The Mel-frequency cepstral coefficient is input into the respiratory sound recognition model to obtain a determination result output by the respiratory sound recognition model, wherein the determination result includes at least one of the following: a second predicted age value, a second gender probability, a second probability of being a minor, and a second eigenvector.

9. The method according to claim 8, characterized in that The determining, based on the determination result, whether the user using the terminal device is a minor user includes: Acquire supplementary data, the supplementary data including at least one of the following: image data collected during face recognition, inertial sensor data collected by an inertial sensor, touch data collected by a touch screen, application usage records, and fingerprint data collected during fingerprint authentication; The determination result and the supplementary data are input into a decision model to obtain a recognition result output by the decision model, wherein the recognition result is used to indicate whether the user using the terminal device is a minor.

10. A terminal device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method according to any one of claims 1 to 9.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images