Audio recognition method, apparatus, device, and storage medium

Abstract

The present disclosure relates to an audio recognition method and device, equipment and storage medium, and belongs to the technical field of computer, which is used to solve the problem of low efficiency of audio recognition model in general technology. The audio recognition method comprises: obtaining to-be-recognized audio data; inputting the to-be-recognized audio data into a target recognition model trained in advance to obtain a recognition result; the target recognition model comprises a plurality of target audio recognition modules; the target audio recognition module is used to obtain a target verification result by verifying the input audio data based on a target verification unit corresponding to the target audio recognition module; the target verification result is used to represent whether to skip the identification processing of the input audio data by the target audio recognition module; the input audio data is audio feature data based on the to-be-recognized audio data; and the target verification unit is used to verify whether the audio signal feature of the input audio data meets a target feature range.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an audio recognition method, apparatus, device, and storage medium. Background Technology

[0002] Dialect recognition technology refers to identifying specific dialect categories from human speech, and it is typically applied at the front end of speech processing systems. Examples include automatic speech recognition (ASR), multilingual translation systems, and biometric authentication. Currently, mainstream dialect recognition technologies train deep learning models in a supervised manner, and then use the trained models to determine the language or dialect attributes of audio. These models usually process audio data using multiple pre-defined processing modules.

[0003] However, the quality of audio data varies greatly. Some audio data is clear and easy to distinguish, while others are difficult to differentiate due to background noise. Applying the same processing procedure to all this data can easily lead to a waste of computing and storage resources, and is also inefficient. Summary of the Invention

[0004] This disclosure provides an audio recognition method, apparatus, device, and storage medium to address the problem of low efficiency in audio recognition models in general technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an audio recognition method is provided, comprising: acquiring audio data to be recognized; inputting the audio data to be recognized into a pre-trained target recognition model to obtain a recognition result; the target recognition model includes a plurality of target audio recognition modules; the target audio recognition modules are used to verify the input audio data based on a target verification unit corresponding to the target audio recognition module to obtain a target verification result; the target verification result is used to characterize whether the recognition processing of the input audio data by the target audio recognition module is skipped; the input audio data is audio feature data based on the audio data to be recognized; the target verification unit is used to verify whether the audio signal features of the input audio data conform to a target feature range.

[0006] Optionally, the target audio recognition module is also used to input the input audio data to the next audio recognition module that is adjacent to and after the target audio recognition module when the target verification result indicates that the target audio recognition module has skipped the recognition processing of the input audio data;

[0007] Alternatively, if the target verification result does not skip the target audio recognition module's recognition processing of the input audio data, the input audio data is processed to obtain output audio data, and the output audio data is input to the next audio recognition module that is adjacent to and follows the target audio recognition module.

[0008] Optionally, the audio recognition method further includes: acquiring multiple sample audio data and an initial recognition model; the initial recognition model includes multiple initial audio recognition modules; different initial audio recognition modules are used to perform different audio recognition tasks; performing an update operation on preset type modules among the multiple initial audio recognition modules to obtain an updated recognition model; the update operation is used to add initial verification units to the preset type modules to obtain an initial target audio recognition module; the initial verification unit is used to verify whether the audio data input to the initial target audio recognition module conforms to the initial feature range; training the updated recognition model according to a preset loss function and multiple sample audio data to obtain a target recognition model; the preset loss function is generated based on the expected pass rate; the expected pass rate is used to represent the proportion of preset type modules that are expected to be skipped among the multiple preset type modules included in the updated recognition model.

[0009] Optionally, the preset type module includes at least one of a convolution type module, an activation type module, and a residual type module; the method of performing an update operation on the preset type modules in multiple initial audio recognition modules to obtain an updated recognition model specifically includes: determining the preset type modules in multiple initial audio recognition modules as recognition modules to be updated, thus obtaining multiple recognition modules to be updated; updating the processing logic of each recognition module to be updated to execute the processing process of the initial verification unit and the audio recognition task corresponding to the recognition module to be updated in parallel, thus obtaining an updated recognition model including multiple initial target audio recognition modules; the initial target audio recognition module is used to verify the audio data to be processed based on the initial verification unit to obtain an initial verification result, and to perform recognition processing on the audio data to be processed to obtain processed audio data, and to perform a weighted summation of the audio data to be processed and the processed audio data based on the initial verification result to obtain an output result, and inputting the output result into the next audio recognition module adjacent to and following the initial target audio recognition module.

[0010] Optionally, the method for training the updated recognition model based on a preset loss function and multiple sample audio data to obtain a target recognition model specifically includes: obtaining a sample subset including a preset number of sample audio data; the sample subset is obtained by partitioning multiple sample audio data; inputting the sample subset into the updated recognition model for recognition processing, and determining the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition processing; determining the loss value corresponding to the sample subset based on the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition processing, the total number of initial target audio recognition modules, and the expected pass rate; when the loss value is less than or equal to a preset threshold, updating the processing logic of each initial target audio recognition module to the process of serially executing the initial verification unit and the audio recognition task corresponding to the initial target audio recognition module, thereby obtaining a target recognition model including multiple target audio recognition modules.

[0011] Optionally, the method for obtaining the audio data to be recognized specifically includes: receiving content data to be recognized sent by a terminal; the content data to be recognized includes at least one segment of speech content; and splicing at least one segment of speech content in the content data to be recognized to obtain the audio data to be recognized.

[0012] Optionally, the method for splicing at least one segment of speech content in the content data to be recognized to obtain the audio data to be recognized specifically includes: determining the start time and end time corresponding to at least one segment of speech content respectively; splicing based on the start time and end time corresponding to each segment of speech content to obtain the audio data to be recognized.

[0013] Optionally, after inputting the audio data to be recognized into a pre-trained target recognition model and obtaining the recognition result, the audio recognition method further includes: determining the content data to be recommended corresponding to the recognition result; and sending the content data to be recommended to the terminal.

[0014] According to a second aspect of the present disclosure, an audio recognition device is provided, comprising: an acquisition unit and a processing unit; the acquisition unit is configured to acquire audio data to be recognized; the processing unit is configured to input the audio data to be recognized into a pre-trained target recognition model to obtain a recognition result; the target recognition model includes a plurality of target audio recognition modules; the target audio recognition modules are used to verify the input audio data based on a target verification unit corresponding to the target audio recognition module to obtain a target verification result; the target verification result is used to characterize whether the recognition processing of the input audio data by the target audio recognition module is skipped; the input audio data is audio feature data based on the audio data to be recognized; the target verification unit is used to verify whether the audio signal features of the input audio data conform to the target feature range.

[0015] Optionally, the target audio recognition module is also used to input the input audio data to the next audio recognition module that is adjacent to and after the target audio recognition module when the target verification result indicates that the target audio recognition module has skipped the recognition processing of the input audio data;

[0016] Alternatively, if the target verification result does not skip the target audio recognition module's recognition processing of the input audio data, the input audio data is processed to obtain output audio data, and the output audio data is input to the next audio recognition module that is adjacent to and follows the target audio recognition module.

[0017] Optionally, the acquisition unit is further configured to acquire multiple sample audio data and an initial recognition model; the initial recognition model includes multiple initial audio recognition modules; different initial audio recognition modules are used to perform different audio recognition tasks; the processing unit is further configured to perform an update operation on preset type modules among the multiple initial audio recognition modules to obtain an updated recognition model; the update operation is used to add an initial verification unit to the preset type modules to obtain an initial target audio recognition module; the initial verification unit is used to verify whether the audio data input to the initial target audio recognition module conforms to the initial feature range; the processing unit is further configured to train the updated recognition model according to a preset loss function and multiple sample audio data to obtain a target recognition model; the preset loss function is generated based on the expected pass rate; the expected pass rate is used to represent the proportion of preset type modules that are expected to be skipped among the multiple preset type modules included in the updated recognition model.

[0018] Optionally, the preset type module includes at least one of a convolution type module, an activation type module, and a residual type module; the processing unit is further configured to perform: determining the preset type module among the multiple initial audio recognition modules as the recognition module to be updated, thereby obtaining multiple recognition modules to be updated; updating the processing logic of each recognition module to be updated to execute the processing process of the audio recognition task corresponding to the initial verification unit and the recognition module to be updated in parallel, thereby obtaining an updated recognition model including multiple initial target audio recognition modules; the initial target audio recognition module is used to verify the audio data to be processed based on the initial verification unit to obtain an initial verification result, and to perform recognition processing on the audio data to be processed to obtain processed audio data, and to perform a weighted summation of the audio data to be processed and the processed audio data based on the initial verification result to obtain an output result, and inputting the output result into the next audio recognition module adjacent to and following the initial target audio recognition module.

[0019] Optionally, the processing unit is further configured to perform: acquiring a sample subset including a preset number of sample audio data; the sample subset is obtained by partitioning multiple sample audio data; inputting the sample subset into the updated recognition model for recognition processing, determining the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition processing; determining the loss value corresponding to the sample subset based on the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition processing, the total number of initial target audio recognition modules, and the expected pass rate; when the loss value is less than or equal to a preset threshold, updating the processing logic of each initial target audio recognition module to serially execute the processing of the initial verification unit and the audio recognition task corresponding to the initial target audio recognition module, thereby obtaining a target recognition model including multiple target audio recognition modules. Optionally, the acquisition unit is further configured to perform: receiving content data to be recognized sent by the terminal; the content data to be recognized includes at least one segment of speech content; splicing at least one segment of speech content in the content data to be recognized to obtain audio data to be recognized.

[0020] Optionally, the processing unit is specifically configured to perform: determining the start time and end time corresponding to at least one segment of speech content; and concatenating the segments of speech content based on their start and end times to obtain the audio data to be recognized.

[0021] Optionally, the audio recognition device further includes: a determining unit; the determining unit is configured to determine the content data to be recommended corresponding to the recognition result; and send the content data to be recommended to the terminal.

[0022] According to a third aspect of the present disclosure, an electronic device is provided, which may include: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional audio recognition methods of the first aspect described above.

[0023] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform any of the optional audio recognition methods of the first aspect described above.

[0024] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions that, when executed on a processor of an electronic device, cause the electronic device to perform the audio recognition method as described in any of the optional implementations of the first aspect.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0026] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0027] Based on any of the above, in this disclosure, after acquiring the audio data to be recognized, the server can input the audio data to be recognized into a pre-trained target recognition model to obtain the recognition result. Since the target recognition model includes multiple target audio recognition modules, each target audio recognition module can verify the input audio data based on its corresponding target verification unit to obtain a target verification result. This target verification result can be used to characterize whether to skip the recognition processing of the input audio data. Therefore, the target recognition model in this disclosure can skip the recognition processing of one or more target audio recognition modules during the recognition processing of the audio data to be recognized, based on the target verification results corresponding to each target audio recognition module. Furthermore, the target feature ranges in the target verification units corresponding to different target audio recognition modules may also be different. Therefore, when determining the recognition result of the audio data to be recognized through the target recognition model, this disclosure can dynamically adjust the number of recognition modules participating in the recognition processing according to the features possessed by the audio data to be recognized, so as to determine the recognition result of the audio data to be recognized based on the model depth most suitable for the audio data to be recognized, thereby avoiding model overfitting and improving model performance.

[0028] In this way, compared to the general approach of processing audio data based on multiple predetermined processing modules, this disclosure can determine the recognition result of the audio data to be recognized based on the model depth most suitable for the audio data to be recognized. Furthermore, since the time spent verifying whether the audio data conforms to the feature range is much less than the time spent on recognition processing, this disclosure can significantly improve the computational efficiency of the target recognition model and save online deployment resources. Therefore, this disclosure can solve the problem of low model processing efficiency in general technologies. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 A schematic diagram of the structure of an audio recognition system provided in an embodiment of this disclosure is shown;

[0031] Figure 2 This illustration shows a structural diagram of a terminal provided in an embodiment of the present disclosure. Figure 1 ;

[0032] Figure 3 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 1 ;

[0033] Figure 4 This illustration shows a structural diagram of an identification module provided in an embodiment of the present disclosure. Figure 1 ;

[0034] Figure 5 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 2 ;

[0035] Figure 6 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 3 ;

[0036] Figure 7 This illustration shows a structural diagram of an identification module provided in an embodiment of the present disclosure. Figure 2 ;

[0037] Figure 8 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 4 ;

[0038] Figure 9 This illustration shows a structural diagram of an identification module provided in an embodiment of the present disclosure. Figure 3 ;

[0039] Figure 10 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 5 ;

[0040] Figure 11 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 6 ;

[0041] Figure 12 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 7 ;

[0042] Figure 13 This illustration shows a flowchart of an audio recognition method provided in an embodiment of the present disclosure. Figure 8 ;

[0043] Figure 14 A schematic diagram of the structure of an audio recognition device provided in an embodiment of this disclosure is shown;

[0044] Figure 15 This illustration shows a structural diagram of a terminal provided in an embodiment of the present disclosure. Figure 2 ;

[0045] Figure 16A schematic diagram of the structure of a server provided in an embodiment of this disclosure is shown. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0047] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0048] It should be noted that the user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0049] Dialect recognition technology refers to identifying specific dialect categories from human speech, and it is typically used in the front end of speech processing systems. Examples include Automatic Speech Recognition (ASR), multilingual translation systems, and biometric authentication. Currently, mainstream dialect recognition technology uses supervised deep learning models to train and then uses these models to determine the language or dialect attributes of audio, resulting in a more comprehensive understanding and description of audio or video. Dialect recognition technology also has important applications in current social content platforms. By extracting dialect information from short videos and live streams, algorithm engineers can improve the personalized recommendation capabilities of social content platforms, more accurately connecting short videos or live streams with users, increasing user engagement and activity. Furthermore, in risk control and related business operations, dialect recognition technology can help identify high-risk short videos and live streams, improving the efficiency of review personnel.

[0050] Traditional dialect recognition techniques rely on Gaussian mixture model-universal background model (GMM-UBM) statistical methods to determine the language or dialect of speech. This method is simple and fast but has poor performance. With the rise of deep learning and the generation of large-scale audio data, time-delay neural networks (TDNNs) have gradually become mainstream, demonstrating strong language recognition capabilities. Currently, various improved versions of TDNNs are emerging, with the ECAPA-TDNN model being a representative example. This model incorporates a Res2Net structure and attention mechanism, significantly improving the model's language discrimination performance.

[0051] However, these models typically process audio data using multiple pre-defined processing modules. But audio data quality varies greatly; some audio data is clear and easy to distinguish, while others have noisy backgrounds and are difficult to differentiate. Applying the same processing flow to all this data can easily lead to a waste of computing and storage resources, and is also inefficient.

[0052] Based on this, embodiments of this disclosure provide an audio recognition method. After acquiring the audio data to be recognized, the server can input the audio data to be recognized into a pre-trained target recognition model to obtain a recognition result. Since the target recognition model includes multiple target audio recognition modules, each target audio recognition module can verify the input audio data based on its corresponding target verification unit to obtain a target verification result. This target verification result can be used to characterize whether to skip the recognition processing of the input audio data. Therefore, the target recognition model in this disclosure can skip the recognition processing of one or more target audio recognition modules during the recognition processing of the audio data to be recognized, based on the target verification results corresponding to each target audio recognition module. Furthermore, the target feature ranges in the target verification units corresponding to different target audio recognition modules may also be different. Therefore, when determining the recognition result of the audio data to be recognized through the target recognition model, this disclosure can dynamically adjust the number of recognition modules participating in the recognition processing according to the features possessed by the audio data to be recognized, so as to determine the recognition result of the audio data to be recognized based on the model depth most suitable for the audio data to be recognized, thereby avoiding model overfitting and improving model performance.

[0053] In this way, compared to the general approach of processing audio data based on multiple predetermined processing modules, this disclosure can determine the recognition result of the audio data to be recognized based on the model depth most suitable for the audio data to be recognized. Furthermore, since the time spent verifying whether the audio data conforms to the feature range is much less than the time spent on recognition processing, this disclosure can significantly improve the computational efficiency of the target recognition model and save online deployment resources. Therefore, this disclosure can solve the problem of low model processing efficiency in general technologies.

[0054] Figure 1 This is a schematic diagram of an audio recognition system provided in an embodiment of the present disclosure, such as... Figure 1 As shown, the audio recognition system 100 may include a server 101 and a terminal 102. The server 101 and the terminal 102 may communicate with each other via a wired network or a wireless network.

[0055] Figure 1 The server 101 can be configured with resource files for a target recognition model used to recognize audio data, so as to realize the audio recognition function.

[0056] In one possible approach, staff can pre-configure resource files containing data such as the initial recognition model, preset loss function, and multiple sample audio data on server 101.

[0057] In one possible approach, server 101 may also include or be connected to a database, and resource files such as the target recognition model and initial recognition model used for recognizing audio data in this disclosure may be stored in the database.

[0058] Optionally, Figure 1 The server 101 can be a single server, or it can be a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. This disclosure does not limit the specific implementation of the server.

[0059] Optionally, Figure 1The terminal 102 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., that can install and use content community applications. This disclosure does not impose any special restrictions on the specific form of the terminal. It can interact with users through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting device.

[0060] Optionally, the above Figure 1 In the audio recognition system shown, server 101 can communicate with at least one terminal 102. This disclosure does not limit the number or type of terminals 102.

[0061] The audio recognition method provided in this disclosure can be applied to the aforementioned... Figure 1 Server 101 in the application scenario shown.

[0062] Combination Figure 1 ,like Figure 2 The diagram shown is a structural schematic of a terminal 102 provided in an embodiment of this disclosure. The terminal 102 may be configured with an input module 21, a display module 22, and a communication module 23. The input module 21 may be a computer external input device such as a mouse and keyboard. The display module 22 may be a device such as a liquid crystal display (LCD) screen, used to display audio data, etc. The communication module 23 may be a transceiver or similar device, used for communication between the terminal 102 and the server 101.

[0063] The audio recognition method provided in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0064] like Figure 3 As shown, when the audio recognition method is applied to a server, the audio recognition method may include: S301-S302.

[0065] S301, The server obtains the audio data to be recognized.

[0066] In one possible approach, the audio data to be recognized can be a segment of audio data of a preset duration. For example, the audio data to be recognized could be a 5-second speech. Alternatively, the audio data to be recognized could also be a 5-second piece of music.

[0067] In one possible approach, the server can extract the audio data to be identified from specific video data or specific audio data. Specific video data could be video data such as game video data, sports video data, or rap video data. Specific audio data could be audio data such as audiobook video data or news video data.

[0068] Combination Figure 2 When it is necessary to identify specific video or audio data to determine its type (e.g., dialect, language, or music genre), the user can perform a confirmation operation (e.g., clicking a confirmation button) through the input module configured on the terminal. In response to the user's confirmation operation, the terminal can send the specific video or audio data to the server. The server can then receive the specific video or audio data.

[0069] Alternatively, while a user is browsing specific video data or listening to specific audio data through a social content service client configured on their terminal, the terminal can send the specific video data or audio data to the server in real time. Correspondingly, the server can receive the specific video data or audio data.

[0070] Next, the server can identify one or more segments of audio content from specific video or audio data, and determine the start and end times of each segment. Then, based on these start and end times, the server can extract these segments, process them into audio streams, and further concatenate them to obtain audio data of a preset duration to be recognized.

[0071] S302. The server inputs the audio data to be recognized into the pre-trained target recognition model to obtain the recognition result.

[0072] In one possible approach, staff can pre-configure a target recognition model for the server to identify the dialect or language type of the audio data.

[0073] The target recognition model may include multiple target audio recognition modules. Furthermore, in addition to multiple target audio recognition modules, the target recognition model may also include multiple initial audio recognition modules. These multiple target audio recognition modules and multiple initial audio recognition modules can be used to perform different audio recognition tasks, that is, to apply different recognition processes to the input audio data. For example, recognition processes based on different parameters such as denoising, transformation, and smoothing.

[0074] It should be noted that when the initial audio recognition module receives input audio data, it can directly process the input audio data to obtain output audio data, and then input the output audio data to the next audio recognition module adjacent to and following the initial audio recognition module. That is, the next initial audio recognition module or the target audio recognition module.

[0075] like Figure 4 As shown, unlike the initial audio recognition module, the target audio recognition module can be used to verify the input audio data based on the target verification unit corresponding to the target audio recognition module to obtain a target verification result. The target verification result can be used to indicate whether the target audio recognition module skips the recognition processing of the input audio data. Furthermore, the target audio recognition module can also be used to input the input audio data to the next audio recognition module adjacent to and following the target audio recognition module when the target verification result indicates that the recognition processing of the input audio data by the target audio recognition module is skipped; or, when the target verification result indicates that the recognition processing of the input audio data by the target audio recognition module is not skipped, to perform recognition processing on the input audio data to obtain output audio data, and to input the output audio data to the next audio recognition module adjacent to and following the target audio recognition module.

[0076] In other words, the target audio recognition module first verifies the input audio data based on the corresponding target verification unit. If the input audio data passes the verification of the target verification unit, the target audio recognition module will not perform the recognition processing of the input audio data. If the input audio data fails the verification of the verification unit, the target audio recognition module will perform the recognition processing of the input audio data.

[0077] In one possible approach, the target verification unit can be used to verify whether the audio signal features of the input audio data conform to the target feature range. Furthermore, if the audio signal features of the input audio data conform to the target feature range, it indicates that the input audio data is relatively simple or has minimal content and is easy to identify, thus requiring no recognition processing from the target audio recognition module. If the audio signal features of the input audio data do not conform to the target feature range, it indicates that the input audio data is relatively complex or has a lot of content and is difficult to identify, thus requiring recognition processing from the target audio recognition module.

[0078] Optionally, the audio signal characteristics of the input audio data can be characteristics such as wavelength, bandwidth, and number of harmonics. When the audio signal characteristic is wavelength, the target characteristic range can be a wavelength range (e.g., 10 cm to 15 cm). When the audio signal characteristic is bandwidth, the target characteristic range can be a bandwidth range (e.g., 5 cm to 10 cm). When the audio signal characteristic is the number of harmonics, the target characteristic range can be a harmonic number range (e.g., 10 to 50 harmonics). This disclosure does not limit these aspects.

[0079] Furthermore, the target feature ranges possessed by the target verification units in different target audio recognition modules can be the same or different. For example, a preset target recognition model includes a first target audio recognition module and a second target audio recognition module, and in the target verification unit corresponding to the first target audio recognition module, the target feature range can be the number of harmonics in the input audio data ranging from 20 to 50. Then, in the target verification unit corresponding to the second target audio recognition module, the target feature range can also be the number of harmonics in the input audio data ranging from 20 to 50. Alternatively, in the target verification unit corresponding to the second target audio recognition module, the target feature range can also be the number of harmonics in the input audio data ranging from 40 to 50.

[0080] Based on this, if the audio data to be identified conforms to the target feature range corresponding to one or more target audio recognition modules, the target recognition model can skip one or more target audio recognition modules during the processing of the audio data to be identified, thereby saving the time consumed in the entire processing process.

[0081] In one possible approach, the input audio data can be audio feature data based on the audio data to be recognized. Specifically, when the input audio data is the audio data input to the first recognition module in the target recognition model arranged according to processing logic, the input audio data is the audio data to be recognized. When the input audio data is the audio data input to other recognition modules in the target recognition model besides the first recognition module, the input audio data can be the audio data of the audio data to be recognized after being processed by one or more recognition modules.

[0082] In one possible approach, after acquiring the audio data to be recognized, the server can input the audio data into a pre-trained target recognition model to obtain the recognition result. Subsequently, the server can send video or audio data in the same dialect or language to the terminal, allowing the terminal to display this data to the user, thereby better meeting user preferences and improving the user experience.

[0083] In one possible example, when the target recognition model is used to identify the dialect type of audio data, the recognition result can be used to indicate that the dialect type of the audio data to be identified is the dialect of a specific location (e.g., dialect A or dialect B). When the target recognition model is used to identify the dialect type of audio data, the recognition result can be used to indicate that the language type of the audio data to be identified is a specific language (e.g., Chinese or English).

[0084] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S301-S302, after the server obtains the audio data to be recognized, it can input the audio data to be recognized into the pre-trained target recognition model to obtain the recognition result. Since the target recognition model includes multiple target audio recognition modules, the target audio recognition modules can verify the input audio data based on the target verification unit corresponding to the target audio recognition module to obtain the target verification result, and the target verification result can be used to characterize whether to skip the recognition processing of the input audio data. Therefore, the target recognition model in this disclosure can skip the recognition processing of one or more target audio recognition modules based on the target verification result corresponding to each target audio recognition module during the recognition processing of the audio data to be recognized. Furthermore, the target feature range in the target verification unit corresponding to different target audio recognition modules may also be different. Therefore, when this disclosure determines the recognition result of the audio data to be recognized through the target recognition model, it can dynamically adjust the number of recognition modules participating in the recognition processing according to the features of the audio data to be recognized, so as to determine the recognition result of the audio data to be recognized based on the model depth most suitable for the audio data to be recognized, thereby avoiding the model overfitting problem and improving the model performance.

[0085] In this way, compared to the general approach of processing audio data based on multiple predetermined processing modules, this disclosure can determine the recognition result of the audio data to be recognized based on the model depth most suitable for the audio data to be recognized. Furthermore, since the time spent verifying whether the audio data conforms to the feature range is much less than the time spent on recognition processing, this disclosure can significantly improve the computational efficiency of the target recognition model and save online deployment resources. Therefore, this disclosure can solve the problem of low model processing efficiency in general technologies.

[0086] In one embodiment, such as Figure 5 As shown, the audio recognition method provided in this disclosure further includes: S401-S403, used to train the target recognition model involved in S302 above.

[0087] S401, The server acquires multiple sample audio data and an initial recognition model.

[0088] The initial recognition model comprises multiple initial audio recognition modules. Different initial audio recognition modules are used to perform different audio recognition tasks. These include, for example, recognition tasks based on different parameters such as denoising, transforming, and smoothing.

[0089] It should be noted that the difference between sample audio data and audio data to be identified lies in their specific content, but they are similar in type and duration. For example, when the audio data to be identified is speech of a preset duration, the sample audio data is also speech of a preset duration.

[0090] In one possible approach, the server can store multiple video data sets and multiple audio data sets. The video data can be, for example, game videos, sports videos, or rap videos. The audio data can be, for example, audio from novels or news videos. Based on this, the server can extract multiple sample audio data sets from the stored multiple video and / or audio data sets.

[0091] It should be understood that the method by which the server extracts multiple sample audio data from multiple video data and / or multiple audio data can be referred to the specific description in S301 of the server extracting audio data to be identified from specific video data or specific audio data, which will not be repeated here.

[0092] In one possible approach, the server can be configured with a storage module, which can pre-store resource files for the initial recognition model. Based on this, the server can obtain the initial recognition model by reading the resource files stored in the storage module. Alternatively, staff can send the resource files for the initial recognition model to the server via a terminal. Correspondingly, the server can receive the resource files for the initial recognition model sent by the terminal to obtain the initial recognition model.

[0093] Optionally, the initial recognition model can be a TDNN-based audio data recognition model, such as the ECAPA-TDNN audio data recognition model. Alternatively, the initial recognition model can also be based on audio data recognition models using other algorithms or architectures. This disclosure does not impose any limitations on this.

[0094] S402. The server performs an update operation on the preset type module among multiple initial audio recognition modules to obtain the updated recognition model.

[0095] In one possible approach, the update operation can be used to add an initial verification unit to a preset type module to obtain an initial target audio recognition module. The initial verification unit is used to verify whether the audio data input to the initial target audio recognition module conforms to the initial feature range.

[0096] In one possible approach, the types of preset type modules can be pre-set in the server by staff based on experience. Examples include convolutional types, activation types, and residual types. Since preset type modules are generally modules repeatedly stacked in the intermediate layers of the model, they typically do not change key information such as feature map size and dimensions. Therefore, for simple audio data, skipping the recognition processing of these modules will not have a substantial impact on the recognition results.

[0097] Optionally, the preset type module can be a convolutional type module, an activation type module, or a residual type module. Alternatively, it can be a batch normalization (BN) layer module, etc. This disclosure does not limit this aspect.

[0098] It should be noted that multiple initial target audio recognition modules obtained by updating multiple preset type modules can have the same initial verification unit. That is, the initial verification units of each initial target audio recognition module can verify the input audio data based on the same initial feature range. Subsequently, during the server's training of the updated recognition model, the initial feature range configured in the initial verification unit can continuously change and converge based on the loss function until it is fixed as the target feature range at the end of training.

[0099] In one possible approach, staff can pre-configure resource files on the server to implement update operations. Based on this, after obtaining the initial recognition model, the server can add initial verification units to preset type modules among multiple initial audio recognition modules according to the update operation resource files, thereby obtaining the updated recognition model.

[0100] In one possible approach, staff can also edit the initial recognition model in real time through the input module configured on the terminal, adding initial verification units to preset type modules in multiple initial audio recognition modules to obtain an updated recognition model.

[0101] S403. The server trains the updated recognition model based on the preset loss function and multiple sample audio data to obtain the target recognition model.

[0102] It should be noted that the preset loss function can be generated based on the expected pass rate. The expected pass rate can be used to represent the percentage of preset type modules that are expected to be skipped among the multiple preset type modules included in the updated recognition model. To avoid skipping too many initial target audio recognition modules during training, which would lead to a decrease in recognition accuracy, it is possible to set the expected pass rate as a percentage of initial target audio recognition modules that can be skipped, and further set the preset loss function based on the expected pass rate, so that the updated recognition model gradually converges to the specified expected pass rate through learning. For example, the preset loss function can be the following function:

[0103]

[0104] Where D is the loss value. M is the number of initial target audio recognition modules skipped during the processing, i.e., the number of initial target audio recognition modules skipped during the recognition process of the updated recognition model to obtain the recognition result, and is a positive integer. N is the total number of multiple initial target audio recognition modules, and is a positive integer. r is the expected pass rate.

[0105] In one possible approach, after obtaining the updated recognition model, the server can configure a preset loss function for the updated recognition model. Then, the server can input multiple batches of sample audio data into the updated recognition model to train the target recognition model.

[0106] Optionally, staff can also pre-set other hyperparameters on the server for training the updated recognition model. For example, the number of latent factors in the matrix factorization. This application does not impose limitations on this.

[0107] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S401-S403, after the server obtains multiple sample audio data and an initial recognition model, it can perform an update operation on the preset type modules in the multiple initial audio recognition modules to obtain an updated recognition model. Furthermore, it can train the updated recognition model based on a preset loss function and multiple sample audio data to obtain a target recognition model. Since the updated recognition model includes multiple initial target audio recognition modules with initial verification units, this disclosure, after training the updated recognition model based on multiple sample audio data and a preset loss function, can obtain a target recognition model including multiple target audio recognition modules. The target audio recognition modules can verify the audio data based on the trained target verification units, thereby dynamically determining whether the recognition processing of the target audio recognition module can be skipped, avoiding model overfitting and improving model performance.

[0108] In one embodiment, combined with Figure 5In S402 above, when the server performs an update operation on the preset type module among multiple initial audio recognition modules to obtain the updated recognition model, such as... Figure 6 As shown, this disclosure provides an optional implementation, including: S501-S502.

[0109] S501. The server identifies the preset type module among multiple initial audio recognition modules as the recognition module to be updated, thus obtaining multiple recognition modules to be updated.

[0110] In one possible approach, the server can determine whether multiple initial audio recognition modules are preset type modules, and identify the initial audio recognition modules that are preset type modules as recognition modules to be updated, thereby obtaining multiple recognition modules to be updated.

[0111] In one possible approach, staff can use a terminal to pre-label multiple initial audio recognition modules in the initial recognition model, assigning tags to preset type modules within each initial audio recognition module and generating corresponding tag files. Furthermore, staff can configure these tag files on a server. Based on this, the server can use the tag files to identify preset type modules from the multiple initial audio recognition modules as the recognition modules to be updated, thus obtaining multiple recognition modules to be updated.

[0112] S502: The server updates the processing logic of each recognition module to be updated to execute the processing of the audio recognition task corresponding to the initial verification unit and the recognition module to be updated in parallel, so as to obtain the updated recognition model including multiple initial target audio recognition modules.

[0113] In one possible approach, after identifying multiple recognition modules to be updated, the server can update the processing logic of each recognition module to execute the processing of the initial verification unit and the audio recognition task corresponding to the recognition module to be updated in parallel, thereby updating each recognition module to be updated to an initial target audio recognition module, resulting in an updated recognition model that includes multiple initial target audio recognition modules.

[0114] It should be noted that the initial target audio recognition module is used to execute in parallel the process of verifying the audio data to be processed based on the initial verification unit to obtain the initial verification result, and recognizing the audio data to be processed to obtain the processed audio data.

[0115] Furthermore, the initial target audio recognition module can also perform a weighted summation of the audio data to be processed and the processed audio data based on the initial verification result to obtain an output result, which is then input into the next audio recognition module adjacent to and following the initial target audio recognition module. Based on this, the initial target audio recognition module can input the audio data to be processed into the next audio recognition module adjacent to and following the initial target audio recognition module if the initial verification result indicates that the initial target audio recognition module should skip its recognition process; or, if the initial verification result indicates that the initial target audio recognition module should not skip its recognition process, the processed audio data is input into the next audio recognition module adjacent to and following the initial target audio recognition module. The audio data to be processed and the processed audio data are audio data associated with the sample audio data.

[0116] In one possible example, such as Figure 7 The diagram shown is a structural schematic of an initial target audio recognition module provided in an embodiment of this disclosure. Figure 7 The initial target audio recognition module is used to execute in parallel the verification process of the initial verification unit on the audio data to be processed, and the process of recognizing and processing the audio data to obtain the processed audio data. Based on this, the initial target audio recognition module does not skip learning from partial sample audio data during training, thus ensuring the training effect of the initial target audio recognition module. This allows the initial target audio recognition module to complete learning from all sample audio data while simultaneously training the initial verification unit.

[0117] and, Figure 7 The initial verification unit can consist of three lightweight operators: a pooling layer, a regularization layer, and a linear layer. Developers can set different parameters or calculation formulas for the pooling, regularization, and linear layers to allow the initial verification unit to verify the audio data based on an initial feature range and output a two-dimensional vector indicating whether the verification passed or failed. For example, a two-dimensional vector of 0 indicates a failed verification (i.e., no), and a vector of 1 indicates a successful verification (i.e., yes).

[0118] Optionally, in Figure 7 In this context, the initial verification unit can also be composed of other lightweight operators capable of achieving the same function. Alternatively, the linear layer in the initial verification unit can be replaced by other more complex nonlinear modules.

[0119] In one possible way, combining Figure 7The output logic of the initial target audio recognition module can be based on the two-dimensional vector output by the initial verification unit, performing a weighted sum of the audio data to be processed and the processed audio data. For example, the weight coefficient of the audio data to be processed can be the two-dimensional vector output by the initial verification unit, and the weight coefficient of the processed audio data can be the difference between 1 and the two-dimensional vector output by the initial verification unit. When the two-dimensional vector is 0 indicating verification failure and 1 indicating verification success, if the two-dimensional vector output by the initial verification unit is 1, then the weight coefficient of the audio data to be processed is 1, and the weight coefficient of the processed audio data is 0. The server then performs a weighted sum of the audio data to be processed and the processed audio data to obtain the audio data to be processed, which is then input into the next audio recognition module. If the two-dimensional vector output by the initial verification unit is 0, then the weight coefficient of the audio data to be processed is 0, and the weight coefficient of the processed audio data is 1. The server then performs a weighted sum of the audio data to be processed and the processed audio data to obtain the processed audio data, which is then input into the next audio recognition module.

[0120] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S501-S502, an optional implementation method is given in which a server performs an update operation on a preset type module among multiple initial audio recognition modules to obtain an updated recognition model. The server can determine the preset type module among multiple initial audio recognition modules as the recognition module to be updated, obtain multiple recognition modules to be updated, and update the processing logic of each recognition module to be updated to execute the processing process of the initial verification unit and the audio recognition task corresponding to the recognition module to be updated in parallel, thereby obtaining an updated recognition model including multiple initial target audio recognition modules. Based on this, in this disclosure, the server can effectively complete the processing of the initial recognition model to obtain the updated recognition model, so as to facilitate the training of the target recognition model.

[0121] In one embodiment, combined with Figure 5 In S403 above, when the server trains the updated recognition model based on a preset loss function and multiple sample audio data to obtain the target recognition model, as follows: Figure 8 As shown, this disclosure provides an optional implementation, including: S601-S604.

[0122] S601. The server obtains a sample subset including a preset number of sample audio data.

[0123] The sample subset is derived from multiple sample audio data. The preset number can be pre-set by staff on the server.

[0124] In one possible approach, to improve the training efficiency of the updated recognition model, the server can input multiple sample audio data into the updated recognition model in batches, thereby training the target recognition model. Specifically, the server can divide the multiple sample audio data into multiple sample subsets according to a rule that each preset number of sample audio data constitutes a sample subset.

[0125] In one possible approach, the server can obtain one of multiple sample subsets and input all the sample audio data from that subset into the updated recognition model at once, so that the updated recognition model can simultaneously process a preset number of sample audio data.

[0126] In one possible approach, when the server inputs a preset number of sample audio data into the updated recognition model at once, the initial verification unit included in the initial target audio recognition module can also simultaneously verify the preset number of sample audio data. In this case, the output of the initial verification unit can be a preset number of two-dimensional vectors corresponding one-to-one with the preset number of sample audio data, indicating whether each sample audio data passes the verification.

[0127] S602. The server inputs the sample subset into the updated recognition model for recognition processing, and determines the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition processing.

[0128] In one possible approach, after obtaining the sample subset, the server can input the sample subset into the updated recognition model for recognition processing. Furthermore, after the updated recognition model has completed the recognition processing of all sample audio data in the sample subset, the server can call the updated recognition model to process the log information of the sample audio data in the sample subset.

[0129] Next, the server can determine the skipping behavior of each initial target audio recognition module when processing the sample subset in the updated recognition model, thereby determining the number of initial target audio recognition modules skipped during the recognition process of the sample audio data in the sample subset.

[0130] S603. The server determines the loss value corresponding to the sample subset based on the number of initial target audio recognition modules skipped during the recognition process of the sample audio data in the sample subset, the total number of initial target audio recognition modules, and the expected pass rate.

[0131] In one possible approach, after the updated recognition model completes the recognition processing of the sample subset, the server can determine the loss value corresponding to the sample subset based on the number of initial target audio recognition modules skipped during the recognition processing of the sample audio data in the sample subset, the total number of initial target audio recognition modules, and the expected pass rate.

[0132] Specifically, based on the formula for the preset loss function shown in S403, the server can determine the number of initial target audio recognition modules skipped during the recognition process (i.e., M) as the ratio of the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition process to a preset number, i.e., the average number of initial target audio recognition modules skipped by the preset number of sample audio data. For all sample subsets, the value of N is the same, i.e., the total number of multiple initial target audio recognition modules in the updated recognition model. Furthermore, based on a preset value of r (e.g., 80%), the server can calculate the value of D, i.e., the loss value corresponding to that sample subset.

[0133] S604. When the loss value is less than or equal to the preset threshold, the server updates the processing logic of each initial target audio recognition module to the process of serially executing the audio recognition task corresponding to the initial verification unit and the initial target audio recognition module, thereby obtaining a target recognition model that includes multiple target audio recognition modules.

[0134] In one possible approach, the preset threshold can be pre-set in the server by staff based on experience to avoid the problem of model overfitting.

[0135] In one possible approach, as the server sequentially inputs multiple sample subsets into the updated recognition model and trains the model, if the loss value corresponding to a sample subset exceeds a preset threshold, it indicates that the sample audio data skipped too many or too few initial target audio recognition modules in the updated recognition model, still failing to meet the pre-set expected pass rate. In this case, the server updates the parameters of the initial verification units in each initial target audio recognition module through backpropagation to reduce the gap between the current actual pass rate and the expected pass rate, bringing the actual pass rate closer to the expected pass rate, thus achieving the training objective of the updated recognition model.

[0136] If the loss value corresponding to a subset of samples is less than or equal to a preset threshold, it indicates that the difference between the actual pass rate and the expected pass rate of the sample audio data in the updated recognition model is small, and it meets the requirements. In this case, to avoid overfitting in the updated recognition model, the server can stop the training process of the updated recognition model and update the processing logic of each initial target audio recognition module to serially execute the audio recognition tasks corresponding to the initial verification unit and the initial target audio recognition module, thereby obtaining a target recognition model that includes multiple target audio recognition modules.

[0137] In one possible example, combining Figure 7 ,like Figure 9 The diagram shown is a structural schematic of another initial target audio recognition module provided in an embodiment of this disclosure. Figure 9 The initial target audio recognition module is used to sequentially execute the verification process of the initial verification unit on the audio data to be processed, and the processing process of the audio recognition task corresponding to the initial target audio recognition module. That is, when the initial target audio recognition module receives the audio data to be processed, it can first verify the audio data based on the initial verification unit. If the audio data passes verification, i.e., it meets the initial feature range, the initial target audio recognition module can directly input the audio data to be processed to the next audio recognition module through the initial verification unit. Alternatively, if the audio data fails verification, i.e., it does not meet the initial feature range, the initial target audio recognition module can perform recognition processing on the audio data to be processed based on a preset type module, and then input the processed audio data to the next audio recognition module.

[0138] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S601-S603, an optional implementation method is given in which the server trains the updated recognition model based on a preset loss function and multiple sample audio data to obtain a target recognition model. The server can train the updated recognition model based on the preset loss function. Since the preset loss function is generated based on the expected pass rate, during the training process, the parameters of each initial verification unit in the initial target audio recognition module will be gradually updated to reduce the gap between the current actual pass rate and the expected pass rate, so that the actual pass rate approaches the expected pass rate, thereby achieving the training purpose of the updated recognition model. Based on this, in this disclosure, the server can update the processing logic of each initial target audio recognition module serially when the loss value is less than or equal to a preset threshold, thereby obtaining a target recognition model including multiple target audio recognition modules.

[0139] In one embodiment, combined with Figure 3 In S301 above, when the server obtains the audio data to be recognized, such as Figure 10As shown, this disclosure provides an optional implementation, including: S701-S702.

[0140] S701, The server receives the data to be identified sent by the terminal.

[0141] The content data to be identified includes at least one segment of audio content. That is, the audio content in the content data to be identified can be continuous or discontinuous.

[0142] Optionally, the content data to be identified can be video data or audio data. Furthermore, the content data to be identified can also include music content and noise content, etc.

[0143] In one possible approach, combining Figure 2 When a user browses content data to be identified (such as video or audio data) through a social content service client configured on their terminal, the terminal can send the content data to be identified to the server in real time. Correspondingly, the server can receive the content data to be identified sent by the terminal.

[0144] S702. The server splices at least one segment of speech content in the data to be recognized to obtain the audio data to be recognized.

[0145] In one possible implementation, after receiving the content data to be recognized from the terminal, the server can identify at least one segment of audio content within the content data and determine the start and end times of each segment. Then, based on the start and end times of each segment, the server can extract at least one segment of audio content. Next, the server can process each segment of audio content into an audio stream and further concatenate them to obtain the audio data to be recognized.

[0146] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S701-S702, an optional implementation method for a server to obtain audio data to be recognized is provided. The server can receive the content data to be recognized sent by the terminal, and splice at least one segment of speech content in the content data to be recognized to obtain the audio data to be recognized. Based on this, this disclosure can support real-time recognition services for the data displayed by the terminal, thereby better recommending content data to users and improving user experience.

[0147] In one embodiment, combined with Figure 10 In the above S702, when the server splices at least one segment of speech content from the data to be recognized to obtain the audio data to be recognized, such as... Figure 11 As shown, this disclosure provides an optional implementation method, including: S801-S802.

[0148] S801, the server determines the start time and end time of at least one segment of audio content.

[0149] In one possible approach, staff can pre-configure a multi-label classification model on the server. This model can be used to classify content in video or audio data and determine the corresponding start and end times. Multi-label classification models can be based on common classification model architectures and training methods, which will not be elaborated upon here.

[0150] Based on this, the server can input the content data to be identified into a multi-label classification model to obtain the classification results of each type of content in the content data to be identified, as well as the corresponding start and end times. Then, the server can select at least one segment of audio content, and the start and end times corresponding to each segment of audio content.

[0151] S802. The server splices together the audio data to be recognized based on the start and end times of each segment of audio content.

[0152] In one possible approach, the server can seamlessly splice together at least one segment of audio content according to its start and end times, in chronological order, to obtain complete audio content. Furthermore, if the audio data to be recognized has a preset duration requirement, the server can extract a segment of audio content of the preset duration from the complete audio content to obtain the audio data to be recognized.

[0153] Based on this, such as Figure 12 As shown, the server can input the content data to be identified into a multi-label classification model to obtain the classification results of each type of content in the content data, along with the corresponding start and end times, i.e., the category and time information of the content data to be identified. Next, the server can select at least one segment of speech content from the category and time information of the content data to be identified, along with the start and end times of each segment. Then, the server can perform concatenation processing based on the start and end times of the at least one segment of speech content to obtain the audio data to be identified. Finally, the server can input the audio data to be identified into a target recognition model to obtain the recognition result. The recognition result can be a multi-dimensional vector. Each dimension of the vector can be used to represent the prediction confidence for a type (e.g., dialect type, language type, or music type). Based on this, the type corresponding to the maximum prediction confidence can be used as the type of the audio data to be identified.

[0154] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S801-S802, an optional implementation method is provided for a server to splice at least one segment of speech content in the content data to be recognized to obtain the audio data to be recognized. The server can determine the start time and end time corresponding to at least one segment of speech content respectively, and splice them based on the start time and end time corresponding to each segment of speech content to obtain the audio data to be recognized. Based on this, the present disclosure can effectively denoise the content data to be recognized to obtain cleaner audio data to be recognized, thereby improving the accuracy of the recognition result.

[0155] In one embodiment, combined with Figure 3 Following S302 above, that is, after the server inputs the audio data to be recognized into the pre-trained target recognition model and obtains the recognition result, as follows: Figure 13 As shown, the audio recognition method provided in this disclosure also includes: S901-S902.

[0156] S901, The server determines the content data to be recommended corresponding to the recognition results.

[0157] In one possible approach, the server can be configured with a storage module. This storage module can be used to store various types of video data and various types of audio data, such as video data in multiple dialects and multiple languages, as well as audio data in multiple dialects, multiple languages, and multiple music genres (e.g., rock and classical).

[0158] In one possible approach, after identifying the audio data to be recognized, the server can read multiple video or audio data corresponding to the recognition result from the storage module, and identify the video or audio data that has been pushed the most times among the multiple video or audio data as the content data to be recommended corresponding to the recognition result. Here, the number of pushes can be used to indicate how many times a video or audio data has been identified as content data to be recommended.

[0159] Alternatively, after reading multiple video or audio data corresponding to the recognition result, the server can also identify the video or audio data whose storage time is closest to the current time as the content data to be recommended corresponding to the recognition result.

[0160] S902, The server sends the data of the content to be recommended to the terminal.

[0161] In one possible approach, after determining the content data to be recommended corresponding to the recognition result, the server can send the content data to be recommended to the terminal. Correspondingly, the terminal can receive the content data to be recommended sent by the server. Then, the terminal can display the content data to be recommended in the configured social content service client, completing the recommendation process to the user.

[0162] The technical solution provided by the above embodiments brings at least the following beneficial effects: As shown in S901-S902, after the server obtains the recognition result corresponding to the audio data to be recognized, the server can also determine the content data to be recommended corresponding to the recognition result and send the content data to be recommended to the terminal. Since the content data to be recommended is similar in data type to the audio data to be recognized, this disclosure can better meet user preferences, improve user experience, and thus increase the number of users and user activity of social content services.

[0163] It is understood that, in actual implementation, the terminal / server described in the embodiments of this disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding audio recognition methods. These hardware structures and / or software modules can constitute an electronic device. Those skilled in the art should readily recognize that, based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0164] Based on this understanding, this disclosure also provides an audio recognition device. Figure 14 A schematic diagram of the structure of the audio recognition device provided in an embodiment of this disclosure is shown. Figure 14 As shown, the audio recognition device may include: an acquisition unit 1001 and a processing unit 1002; the acquisition unit 1001 is configured to acquire audio data to be recognized; the processing unit 1002 is configured to input the audio data to be recognized into a pre-trained target recognition model to obtain a recognition result; the target recognition model includes multiple target audio recognition modules; the target audio recognition modules are used to verify the input audio data based on the target verification unit corresponding to the target audio recognition module to obtain a target verification result; the target verification result is used to characterize whether the recognition processing of the input audio data by the target audio recognition module is skipped; the input audio data is audio feature data based on the audio data to be recognized; the target verification unit is used to verify whether the audio signal features of the input audio data conform to the target feature range.

[0165] Optionally, the target audio recognition module is also used to input the input audio data to the next audio recognition module that is adjacent to and after the target audio recognition module when the target verification result indicates that the target audio recognition module has skipped the recognition processing of the input audio data;

[0166] Alternatively, if the target verification result does not skip the target audio recognition module's recognition processing of the input audio data, the input audio data is processed to obtain output audio data, and the output audio data is input to the next audio recognition module that is adjacent to and follows the target audio recognition module.

[0167] Optionally, the acquisition unit 1001 is further configured to acquire multiple sample audio data and an initial recognition model; the initial recognition model includes multiple initial audio recognition modules; different initial audio recognition modules are used to perform different audio recognition tasks; the processing unit 1002 is further configured to perform an update operation on preset type modules among the multiple initial audio recognition modules to obtain an updated recognition model; the update operation is used to add an initial verification unit to the preset type modules to obtain an initial target audio recognition module; the initial verification unit is used to verify whether the audio data input to the initial target audio recognition module conforms to the initial feature range; the processing unit 1002 is further configured to train the updated recognition model according to a preset loss function and multiple sample audio data to obtain a target recognition model; the preset loss function is generated based on the expected pass rate; the expected pass rate is used to represent the proportion of preset type modules that are expected to be skipped among the multiple preset type modules included in the updated recognition model.

[0168] Optionally, the preset type module includes at least one of a convolution type module, an activation type module, and a residual type module; the processing unit 1002 is further configured to perform: determining the preset type module among the multiple initial audio recognition modules as the recognition module to be updated, thereby obtaining multiple recognition modules to be updated; updating the processing logic of each recognition module to be updated to execute the processing process of the audio recognition task corresponding to the initial verification unit and the recognition module to be updated in parallel, thereby obtaining an updated recognition model including multiple initial target audio recognition modules; the initial target audio recognition module is used to verify the audio data to be processed based on the initial verification unit to obtain an initial verification result, and to perform recognition processing on the audio data to be processed to obtain processed audio data, and to perform a weighted summation of the audio data to be processed and the processed audio data based on the initial verification result to obtain an output result, and inputting the output result into the next audio recognition module adjacent to and following the initial target audio recognition module.

[0169] Optionally, the processing unit 1002 is further configured to perform: acquiring a sample subset including a preset number of sample audio data; the sample subset is obtained by dividing multiple sample audio data; inputting the sample subset into the updated recognition model for recognition processing, determining the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition processing; determining the loss value corresponding to the sample subset based on the number of initial target audio recognition modules skipped by the sample audio data in the sample subset during the recognition processing, the total number of initial target audio recognition modules, and the expected pass rate; when the loss value is less than or equal to a preset threshold, updating the processing logic of each initial target audio recognition module to serially execute the processing of the audio recognition task corresponding to the initial verification unit and the initial target audio recognition module, thereby obtaining a target recognition model including multiple target audio recognition modules. Optionally, the acquisition unit 1001 is specifically configured to perform: receiving content data to be recognized sent by the terminal; the content data to be recognized includes at least one segment of speech content; splicing at least one segment of speech content in the content data to be recognized to obtain audio data to be recognized.

[0170] Optionally, the processing unit 1002 is specifically configured to perform: determining the start time and end time corresponding to at least one segment of speech content; and splicing the segments of speech content based on their start times and end times to obtain the audio data to be recognized.

[0171] Optionally, the audio recognition device further includes: a determination unit 1003; the determination unit 1003 is configured to determine the content data to be recommended corresponding to the recognition result; and send the content data to be recommended to the terminal.

[0172] As described above, the embodiments of this disclosure can divide the audio recognition device into functional modules according to the above method examples. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, it should be noted that the module division in these embodiments is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into a single processing module.

[0173] Regarding the audio recognition device in the above embodiments, the specific methods by which each module performs its operations and the beneficial effects thereof have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0174] This disclosure also provides a terminal, which may be a user terminal such as a mobile phone or a computer. Figure 15A schematic diagram of the structure of a terminal provided in an embodiment of this disclosure is shown. The terminal may be an audio recognition device and may include at least one processor 61, a communication bus 62, a memory 63, and at least one communication interface 64.

[0175] Processor 61 may be a CPU, a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present disclosure.

[0176] The communication bus 62 may include a path for transmitting information between the aforementioned components.

[0177] Communication interface 64 uses any transceiver-like device for communicating with other devices or communication networks, such as servers, Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc.

[0178] Memory 63 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0179] The memory 63 stores the application code that executes the present invention, and its execution is controlled by the processor 61. The processor 61 executes the application code stored in the memory 63 to implement the functions of the method of the present invention.

[0180] In a specific implementation, as one example, processor 61 may include one or more CPUs, for example... Figure 15 CPU0 and CPU1 in the CPU.

[0181] In a specific implementation, as one example, the terminal may include multiple processors, for example... Figure 15 Processors 61 and 65 are included. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0182] In a specific implementation, as one embodiment, the terminal may further include an input device 66 and an output device 67. The input device 66 and output device 67 communicate and can accept user input in various ways. For example, the input device 66 may be a mouse, keyboard, touchscreen device, or sensing device. The output device 67 communicates with the processor 61 and can display information in various ways. For example, the output device 61 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, etc.

[0183] Those skilled in the art will understand that Figure 15 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0184] This disclosure also provides a server. Figure 16 A schematic diagram of the server structure provided in an embodiment of this disclosure is shown. This server can be an audio recognition device. The server can vary significantly due to differences in configuration or performance, and may include one or more processors 71 and one or more memories 72. The memory 72 stores at least one instruction, which is loaded and executed by the processor 71 to implement the audio recognition method provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0185] This disclosure also provides a computer-readable storage medium including instructions stored thereon, which, when executed by a processor of a computer device, enable the computer to perform the audio recognition method provided in the embodiments described above. For example, the computer-readable storage medium may be a memory 63 including instructions, which may be executed by a processor 61 of a terminal to complete the method. As another example, the computer-readable storage medium may be a memory 72 including instructions, which may be executed by a processor 71 of a server to complete the method. Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0186] This disclosure also provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the above-described actions. Figure 3 , Figure 5 , Figure 6 , Figure 8 and Figures 10-13 The audio recognition method shown in any of the attached figures.

[0187] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0188] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio recognition method, characterized in that, include: Obtain the audio data to be recognized; The audio data to be identified is input into a pre-trained target recognition model to obtain the recognition result; The target recognition model includes multiple target audio recognition modules; The target audio recognition module is a module repeatedly stacked in the intermediate layers of the target recognition model, without changing the size and dimension of the feature map; the target audio recognition module is of convolutional, activation, and residual types; the target audio recognition module is used to verify the input audio data based on the target verification unit corresponding to the target audio recognition module to obtain a target verification result; the target verification result is used to indicate whether to skip the recognition processing of the input audio data by the target audio recognition module; the input audio data is audio feature data based on the audio data to be recognized. The target verification unit is used to verify whether the audio signal features of the input audio data conform to the target feature range.

2. The audio recognition method according to claim 1, characterized in that, The target audio recognition module is further configured to input the input audio data to the next audio recognition module adjacent to and following the target audio recognition module when the target verification result indicates that the target audio recognition module has skipped the recognition processing of the input audio data; Alternatively, if the target verification result indicates that the target audio recognition module does not skip the recognition processing of the input audio data, the input audio data is processed to obtain output audio data, and the output audio data is input to the next audio recognition module that is adjacent to and follows the target audio recognition module.

3. The audio recognition method according to claim 1, characterized in that, Also includes: Acquire multiple sample audio data and an initial recognition model; the initial recognition model includes multiple initial audio recognition modules; Different initial audio recognition modules are used to perform different audio recognition tasks; An update operation is performed on the preset type modules among the multiple initial audio recognition modules to obtain an updated recognition model; the update operation is used to add an initial verification unit to the preset type modules to obtain an initial target audio recognition module; The initial verification unit is used to verify whether the audio data input into the initial target audio recognition module conforms to the initial feature range; The updated recognition model is trained based on a preset loss function and multiple sample audio data to obtain the target recognition model; the preset loss function is generated based on the expected pass rate; the expected pass rate is used to represent the proportion of preset type modules that are expected to be skipped among the multiple preset type modules included in the updated recognition model.

4. The audio recognition method according to claim 3, characterized in that, The preset type module includes at least one of a convolution type module, an activation type module, and a residual type module; the step of performing an update operation on the preset type modules in the plurality of initial audio recognition modules to obtain an updated recognition model includes: A preset type module among the multiple initial audio recognition modules is identified as a recognition module to be updated, thus obtaining multiple recognition modules to be updated; The processing logic of each of the recognition modules to be updated is updated to execute the processing of the audio recognition tasks corresponding to the initial verification unit and the recognition module to be updated in parallel, so as to obtain the updated recognition model including multiple initial target audio recognition modules. The initial target audio recognition module is used to verify the audio data to be processed based on the initial verification unit to obtain an initial verification result, and to recognize the audio data to be processed to obtain processed audio data. Based on the initial verification result, the audio data to be processed and the processed audio data are weighted and summed to obtain an output result. The output result is then input to the next audio recognition module that is adjacent to and follows the initial target audio recognition module.

5. The audio recognition method according to claim 4, characterized in that, The step of training the updated recognition model based on a preset loss function and multiple sample audio data to obtain the target recognition model includes: Obtain a sample subset comprising a preset number of the sample audio data; the sample subset is obtained by partitioning based on multiple sample audio data. The sample subset is input into the updated recognition model for recognition processing, and the number of the initial target audio recognition modules that the sample audio data in the sample subset skips during the recognition processing is determined. Based on the number of initial target audio recognition modules skipped during the recognition process, the total number of initial target audio recognition modules, and the expected pass rate, the loss value corresponding to the sample subset is determined. When the loss value is less than or equal to a preset threshold, the processing logic of each initial target audio recognition module is updated to serially execute the processing of the audio recognition task corresponding to the initial verification unit and the initial target audio recognition module, thereby obtaining the target recognition model including multiple target audio recognition modules.

6. The audio recognition method according to claim 1, characterized in that, The acquisition of the audio data to be identified includes: The receiving terminal sends data of content to be identified; the data of content to be identified includes at least one segment of audio content; At least one segment of the speech content in the data to be identified is spliced ​​together to obtain the audio data to be identified.

7. The audio recognition method according to claim 6, characterized in that, The step of splicing at least one segment of the speech content in the data to be identified to obtain the audio data to be identified includes: Determine the start time and end time corresponding to at least one segment of the aforementioned audio content; The audio data to be recognized is obtained by splicing together the start and end times corresponding to the audio content of each segment.

8. The audio recognition method according to claim 1, characterized in that, After inputting the audio data to be recognized into a pre-trained target recognition model to obtain the recognition result, the method further includes: Determine the content data to be recommended corresponding to the recognition result; The data of the content to be recommended is sent to the terminal.

9. An audio recognition device, characterized in that, include: Acquisition unit and processing unit; The acquisition unit is configured to acquire audio data to be recognized; The processing unit is configured to input the audio data to be recognized into a pre-trained target recognition model to obtain a recognition result; the target recognition model includes multiple target audio recognition modules. The target audio recognition module is a module repeatedly stacked in the intermediate layers of the target recognition model, without changing the size and dimension of the feature map; the target audio recognition module is of convolutional, activation, and residual types; the target audio recognition module is used to verify the input audio data based on the target verification unit corresponding to the target audio recognition module to obtain a target verification result; the target verification result is used to indicate whether to skip the recognition processing of the input audio data by the target audio recognition module; the input audio data is audio feature data based on the audio data to be recognized. The target verification unit is used to verify whether the audio signal features of the input audio data conform to the target feature range.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the audio recognition method as described in any one of claims 1-8.

11. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the audio recognition method as described in any one of claims 1-8.

Images