Audio processing method, device, equipment, storage medium and program product

Abstract

The present application provides an audio processing method, device, equipment, storage medium and program product, which relate to the field of audio and video technology. The method includes: obtaining a first audio; performing voice positioning on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment including voice; obtaining the audio features of at least one target audio segment; performing language recognition based on the audio features of at least one target audio segment to obtain the language type of at least one target audio segment; based on the language type of at least one target audio segment, obtaining the language type of the first audio. Through the above method, the manual labeling process in the process of obtaining the language type of the first audio is avoided, the language recognition process is simplified, the efficiency of obtaining the language type of audio and video content is improved, and the accuracy of obtaining the language type is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio and video technology, and in particular to an audio processing method, device, equipment, storage medium and program product. Background Art

[0002] With the development of online platforms, the content in online platforms is gradually increasing. In order to facilitate the management and recommendation of online audio and video, online platforms usually classify and manage online audio and video; for example, audio and video content is classified based on language type.

[0003] In related technologies, when classifying audio and video content, a computer device can use speech recognition technology to identify the voice information contained in the audio and video content, convert the voice information into text information, and then perform language classification based on the text information to obtain the language type of the audio and video content.

[0004] However, in the above process, the language classification based on text needs to be annotated by relevant personnel, which makes the efficiency of obtaining the language type of audio and video content low and the accuracy poor. Summary of the invention

[0005] The embodiments of the present application provide an audio processing method, apparatus, device, storage medium and program product, which can avoid the manual labeling process in the process of obtaining the language type of the first audio, simplify the language recognition process, and thus improve the efficiency of obtaining the language type of audio and video content and the accuracy of obtaining the language type. The technical solution is as follows:

[0006] In one aspect, a method for audio processing is provided, the method comprising:

[0007] Get the first audio;

[0008] Performing voice localization on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment containing voice;

[0009] Acquire audio features of at least one of the target audio segments;

[0010] Performing language recognition based on the audio features of at least one of the target audio segments to obtain the language type of at least one of the target audio segments;

[0011] Based on the language type of at least one of the target audio segments, the language type of the first audio is obtained.

[0012] In another aspect, an audio processing device is provided, the device comprising:

[0013] An audio acquisition module, used to acquire a first audio;

[0014] A speech localization module, configured to perform speech localization on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment containing speech;

[0015] A feature acquisition module, used to acquire audio features of at least one target audio segment;

[0016] A language recognition module, configured to perform language recognition based on the audio features of at least one of the target audio segments, and obtain the language type of each of the at least one of the target audio segments;

[0017] The language acquisition module is used to acquire the language type of the first audio based on the language type of at least one of the target audio segments.

[0018] In a possible implementation, the feature acquisition module is used to process at least one of the target audio segments respectively based on the first feature extraction layer in the language recognition network to obtain audio features of at least one of the target audio segments;

[0019] The language recognition module is used to process the audio features of at least one target audio segment based on the language recognition layer in the language recognition network to obtain the language type of at least one target audio segment;

[0020] The language recognition network is a network obtained by training based on sample audio segments and language labels of the sample audio segments.

[0021] In a possible implementation, the voice localization module includes:

[0022] A voice positioning submodule, configured to perform voice positioning on the first audio and obtain time information of at least one of the target audio segments; the time information is used to indicate the start time and end time of the target audio segment;

[0023] The segment extraction submodule is used to extract at least one target audio segment from the first audio based on the time information of at least one target audio segment.

[0024] In a possible implementation, the voice localization submodule includes:

[0025] a feature acquisition unit, configured to extract audio features from the first audio to obtain audio features of the first audio;

[0026] A time information acquisition unit is used to perform voice localization on the first audio based on the audio features of the first audio, and acquire the time information of at least one of the target audio segments.

[0027] In a possible implementation, the feature acquisition unit is used to:

[0028] Extracting frequency spectrum features of the first audio;

[0029] Based on the frequency spectrum characteristics of the first audio, generating an audio feature graph of the first audio;

[0030] Feature extraction is performed on the audio feature graph of the first audio to obtain audio features of the first audio.

[0031] In a possible implementation, the feature acquisition unit is used to:

[0032] Processing the audio feature graph of the first audio based on an audio localization network to obtain audio features of the first audio;

[0033] The time information acquisition unit is used to process the audio features of the first audio based on the audio classification layer in the audio localization network to obtain the time information of at least one target audio segment;

[0034] The audio localization network is a network obtained by training based on sample audio and audio segment labels of the sample audio.

[0035] In a possible implementation, the language acquisition module is used to count the number of audio segments of each language type in at least one of the target audio segments based on the language type of each of the at least one target audio segments;

[0036] The language type containing the largest number of the target audio segments among the various language types is determined as the language type of the first audio.

[0037] In a possible implementation, the language acquisition module includes:

[0038] A cumulative duration acquisition submodule, configured to acquire the cumulative durations of the target audio segments corresponding to the respective language types based on the language type of at least one of the target audio segments;

[0039] The language acquisition submodule is used to acquire the language type of the first audio based on the accumulated durations corresponding to each language type.

[0040] In a possible implementation, the language acquisition submodule is used to:

[0041] Determine the language type with the longest cumulative duration among the various language types as the language type of the first audio;

[0042] or,

[0043] Among the language types, the language type having the largest ratio between the accumulated duration and the duration of the first audio is determined as the language type of the first audio.

[0044] In a possible implementation, the accumulated duration acquisition submodule includes:

[0045] a position information acquisition unit, configured to acquire position information of at least one of the target audio segments in the first audio; the position information is used to indicate a time position of the target audio segment in the first audio;

[0046] a weighted processing unit, configured to perform weighted processing on the duration of each of at least one of the target audio segments based on the position information of at least one of the target audio segments in the first audio;

[0047] The cumulative duration acquisition unit is used to acquire the cumulative durations of the target audio segments corresponding to each language type based on the language type of at least one of the target audio segments and the weighted duration of at least one of the target audio segments.

[0048] In a possible implementation, the location information acquiring unit is used to:

[0049] Obtaining a reference time point of at least one of the target audio segments in the first audio; the reference time point is one of a starting time point, a middle time point, and an ending time point of the audio segment;

[0050] Based on a reference time point of at least one of the target audio segments in the first audio, respectively, obtaining an offset duration of at least one of the target audio segments as position information of at least one of the target audio segments in the first audio; the offset duration is a duration between the reference time point and an intermediate time point of the first audio;

[0051] The weighted processing unit is used to obtain a weight of at least one of the target audio segments; the weight is inversely correlated with the offset duration;

[0052] Based on the weight of at least one of the target audio segments, weighted processing is performed on the segment duration of at least one of the target audio segments.

[0053] On the other hand, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-mentioned audio processing method.

[0054] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned audio processing method.

[0055] On the other hand, a computer program product is provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor to implement the audio processing method provided in the above-mentioned various optional implementations.

[0056] The technical solution provided by this application may have the following beneficial effects:

[0057] By extracting the target audio segments containing speech in the first audio, the audio features of each target audio segment are obtained, so as to obtain the language type of each target audio segment based on the audio features of each target audio segment, and then, the language type of the first audio is obtained based on the language type of each target audio segment. In the above scheme, by performing language recognition on each audio segment containing speech in the audio, the language of the audio is automatically acquired, thereby avoiding the manual labeling process in the process of acquiring the language type of the first audio, simplifying the language recognition process, improving the efficiency of acquiring the language type of audio and video content, and improving the accuracy of acquiring the language type.

[0058] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] Figure 1 A system structure diagram used in an audio processing method provided by an exemplary embodiment of the present application is shown;

[0061] Figure 2 A flowchart of an audio processing method provided by an exemplary embodiment of the present application is shown;

[0062] Figure 3 is a framework diagram of model training and audio processing according to an exemplary embodiment;

[0063] Figure 4A flowchart of an audio processing method provided by an exemplary embodiment of the present application is shown;

[0064] Figure 5 A flowchart of an audio processing method provided by an exemplary embodiment of the present application is shown;

[0065] Figure 6 A schematic diagram of audio feature extraction based on STFT provided by an exemplary embodiment of the present application is shown;

[0066] Figure 7 A schematic diagram of an audio STFT energy spectrum provided by an exemplary embodiment of the present application is shown;

[0067] Figure 8 A schematic diagram of a MEL filter bank shown in an exemplary embodiment of the present application is shown;

[0068] Fig. 9 A structural diagram of a VGGish-embedding network provided by an exemplary embodiment of the present application is shown;

[0069] Fig.10 A schematic diagram of the structure of an audio classification layer shown in an exemplary embodiment of the present application is shown;

[0070] Fig.11 A schematic diagram showing an audio feature extraction process provided by an exemplary embodiment of the present application is shown;

[0071] Fig.12 A schematic diagram of the structure of a language recognition layer shown in an exemplary embodiment of the present application is shown;

[0072] Fig.13 A schematic diagram of an audio processing process shown in an exemplary embodiment of the present application is shown;

[0073] Fig.14 A schematic diagram showing an application of an audio processing method provided by an exemplary embodiment of the present application is shown;

[0074] Fig.15 A block diagram of an audio processing device exemplarily shown in the present application is shown;

[0075] Fig.16 is a structural block diagram of a computer device according to an exemplary embodiment;

[0076] Fig.17 It is a structural block diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0078] For ease of understanding, the terms involved in this application are explained below.

[0079] 1) Artificial Intelligence (AI)

[0080] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0081] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The display device including an image acquisition component shown in this application mainly involves computer vision technology and machine learning / deep learning.

[0082] 2) Computer Vision (CV)

[0083] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (Three Dimensions) technology, virtual reality, augmented reality and map construction, as well as common biometric recognition technologies such as face recognition.

[0084] Figure 1 FIG. 4 shows a system structure diagram used in an audio processing method provided by an exemplary embodiment of the present application. Figure 1 As shown, the system includes: a server 110 and a terminal 120.

[0085] The above-mentioned server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0086] The terminal 120 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a vehicle-mounted terminal, etc. with video processing capabilities, but is not limited thereto.

[0087] The above system includes one or more servers 110 and multiple terminals 120. The embodiment of the present application does not limit the number of servers 110 and terminals 120.

[0088] The terminal and the server are connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0089] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology. This application is not limited here.

[0090] Figure 2 A flowchart of an audio processing method provided by an exemplary embodiment of the present application is shown. Figure 2 As shown, the method can be executed by a computer device, which can be implemented as a terminal or a server, and the terminal or the server can be as shown in Figure 1 The terminal or server shown, such as Figure 2 As shown, the audio processing method may include the following steps:

[0091] Step 210: Acquire first audio.

[0092] In an embodiment of the present application, the first audio may refer to audio of the language type to be acquired; wherein the first audio may be recorded audio information; or, the first audio may be audio information extracted from other content containing audio elements, for example, the first audio may be audio information extracted from video content, or, the first audio may be audio information extracted from game content, and so on.

[0093] Step 220, perform voice localization on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment including voice.

[0094] The first audio may be composed of at least one audio segment; these audio segments may include target audio segments; different target audio segments in the same audio may have different language types, or different target audio segments may have the same language type.

[0095] Illustratively, the audio segment containing speech can be implemented as lyrics content in a song, an audio segment corresponding to lines content in a video, or an audio segment corresponding to lyrics content in background music in a video, and so on.

[0096] Step 230: Acquire audio features of at least one target audio segment.

[0097] Step 240: Perform language recognition based on the audio features of at least one target audio segment to obtain the language type of each of the at least one target audio segment.

[0098] Step 250: Acquire the language type of the first audio based on the language type of at least one target audio segment.

[0099] In summary, the audio processing method provided in the embodiment of the present application extracts the target audio segments containing speech in the first audio to obtain the audio features of each target audio segment, so as to obtain the language type of each target audio segment based on the audio features of each target audio segment, and then obtains the language type of the first audio based on the language type of each target audio segment; in the above scheme, the language of the audio is automatically acquired by performing language recognition on each audio segment containing speech in the audio, thereby avoiding the manual labeling process in the process of acquiring the language type of the first audio, simplifying the language recognition process, improving the efficiency of acquiring the language type of audio and video content, and improving the accuracy of acquiring the language type.

[0100] In the embodiment of the present application, the target audio segments containing speech in the first audio can be extracted to obtain the language type of at least one target audio segment respectively, and then the language type of the first audio can be obtained based on the language type of at least one target audio segment; thereby avoiding the manual annotation process in the process of obtaining the language type of the first audio, simplifying the language recognition process, improving the efficiency of obtaining the language type of the audio and video content, and improving the accuracy of obtaining the language type; the audio processing method provided by the present application can be used in but not limited to the following scenarios:

[0101] 1. Audio classification scenario.

[0102] In an application with an audio playback function, a function of classifying and displaying audio information is usually provided to improve the display or recommendation effect of the audio information. In this process, the computer device can automatically identify the language type of the audio information in the application through the audio processing method provided in the embodiment of the present application; the audio information in the application can be the audio information uploaded by the uploader with audio upload authority; the process of automatically identifying the language type of the audio information in the application can be implemented as follows: obtaining at least one target audio segment in the uploaded audio information through voice positioning, performing language identification on each target audio segment to obtain the language type of each target audio segment, thereby obtaining the language type of the audio information based on the language type of each target audio segment; the computer device can classify each audio information based on the language type of each audio information; schematically, the audio information in the application can be classified into Chinese songs, English songs, Japanese songs, etc., so as to classify and display or recommend audio information in different languages; through the above method, the accuracy and classification effect of classifying audio information can be improved, thereby improving the display or recommendation effect of audio information.

[0103] 2. Video classification scenarios.

[0104] Video classification refers to classifying the content contained in a given video clip; when classifying video content, the video content can be classified based on the audio information contained in the video; when classifying video content based on the audio information contained in the video, the audio processing method provided in the embodiment of the present application can be used to first obtain the audio information in the video content, and then automatically identify the language type of the extracted audio information to obtain the language type of the audio information, so as to classify the video content according to the language type of the audio information, so as to improve the accuracy and classification effect of classifying the video content.

[0105] The solution involved in this application includes a model training stage and an audio processing stage. Figure 3 is a framework diagram of a model training and audio processing according to an exemplary embodiment. Figure 3As shown, in the model training stage, the model training device 310 trains the language recognition model including the audio localization network and the language recognition network through pre-set training samples (including audio samples and the annotation information of the audio samples, the annotation information of the audio samples can be used to indicate each target audio segment in the audio sample, and the language type of the audio sample) to obtain the language recognition model after training. In the audio processing stage, the audio processing device 320 performs audio processing on the input audio information based on the language recognition model to obtain the language type of each of the at least one target audio segment in the audio information, and then obtains the language type of the input audio information based on the language type of each of the at least one target audio segment.

[0106] The model training device 310 and the audio processing device 320 may be computer devices. For example, the computer device may be Figure 1 The server or terminal shown.

[0107] Optionally, the above-mentioned model training device 310 and audio processing device 320 can be the same device, or the model training device 310 and the audio processing device 320 can also be different devices. Moreover, when the model training device 310 and the audio processing device 320 are different devices, the model training device 310 and the audio processing device 320 can be devices of the same type, such as the model training device 310 and the audio processing device 320 can both be servers; or, the model training device 310 and the audio processing device 320 can also be different types of devices, such as the audio processing device 320 can be a terminal, and the model training device 310 can be a server, etc. The embodiment of the present application does not limit the specific types of the model training device 310 and the audio processing device 320.

[0108] Figure 4 A flowchart of an audio processing method provided by an exemplary embodiment of the present application is shown. Figure 4 As shown, the method can be executed by a computer device, which can be implemented as a terminal or a server, and the terminal or the server can be as shown in Figure 1 The terminal or server shown, such as Figure 4 As shown, the audio processing method may include the following steps:

[0109] Step 410, obtaining an audio sample set, the audio sample set comprising a first audio sample and a second audio sample; the first audio sample has first annotation information for indicating each target audio segment in the first audio sample; the second audio sample has second annotation information for indicating the language type of each target audio segment in the second audio sample.

[0110] Illustratively, the first annotation information may include time information of the target audio segment, and the time information may be used to indicate the start time and end time of each target audio segment in the audio sample.

[0111] Optionally, the time information may include the start and end time of each target audio segment; or, the time information may include the start time of each target audio segment and the duration of each target audio segment, or, the time information may also include the end time and duration of each target audio segment, thereby indicating the start time and end time of each target audio segment.

[0112] Optionally, the first audio sample and the second audio sample may be the same or different; when the first audio sample and the second audio sample are the same, the first audio sample has first annotation information for indicating each target audio segment in the first audio sample, and second annotation information for indicating the language type of each target audio segment in the first audio sample.

[0113] Step 420: Process the first audio sample based on the audio localization network in the language recognition model to obtain at least one predicted audio segment of the first audio sample.

[0114] Step 430: Process at least one target audio segment of the second audio sample based on the language recognition network in the language recognition model to obtain a predicted language type of the at least one target audio segment of the second audio sample.

[0115] Step 440 , training a language recognition model based on at least one predicted audio segment and the first annotation information, and the predicted language type of at least one target audio segment and the second annotation information.

[0116] The trained language recognition model can be used to process the input audio to obtain each target audio segment of the input audio and the language type of each target audio segment, so that the computer device can determine the language type of the input audio based on the language type of each target audio segment.

[0117] In a possible implementation, based on at least one predicted audio segment and the first annotation information, the predicted language type of at least one target audio segment and the second annotation information include:

[0118] Training an audio localization network based on at least one predicted audio segment and the first annotation information;

[0119] The language recognition network is trained based on the predicted language type of at least one target audio segment and the second annotation information.

[0120] After the audio localization network and the language recognition network are trained, a language recognition model is constructed based on the trained audio localization network and the trained language recognition network.

[0121] That is to say, the language recognition model in the embodiment of the present application includes an audio localization network and a language recognition network; the audio localization network and the language recognition network are trained separately.

[0122] Optionally, the process of training the audio localization network based on at least one predicted audio segment and the first annotation information can be implemented as follows:

[0123] Calculating a function value of a first loss function based on at least one predicted audio segment and the first annotation information;

[0124] Parameters of the audio localization network are updated based on the function value of the first loss function.

[0125] The process of training the language recognition network based on the predicted language type of at least one target audio segment and the second annotation information can be implemented as follows:

[0126] Calculating a function value of a second loss function based on the predicted language type of at least one target audio segment and the second annotation information;

[0127] The parameters of the language recognition network are updated based on the function value of the second loss function.

[0128] In summary, the audio processing method provided in the embodiment of the present application trains a language recognition model based on audio samples and annotation information of the audio samples, and the language recognition model includes an audio localization network and a language recognition network; the input audio is processed by the trained language recognition network to obtain the language type of the target audio segment in the input audio, and then, based on the language type of the target audio segment in the input audio, the language type of the input audio is obtained; in the above scheme, the language of the audio is automatically obtained by performing language recognition on each audio segment containing speech in the audio, thereby avoiding the manual annotation process in the process of obtaining the language type of the first audio, simplifying the language recognition process, improving the efficiency of obtaining the language type of audio and video content, and improving the accuracy of obtaining the language type.

[0129] Figure 5 A flowchart of an audio processing method provided by an exemplary embodiment of the present application is shown. Figure 5 As shown, the method can be executed by a computer device, which can be implemented as a terminal or a server, and the terminal or the server can be as shown in Figure 1 The terminal or server shown, such as Figure 5 As shown, the audio processing method may include the following steps:

[0130] Step 510: Acquire first audio.

[0131] In the embodiment of the present application, the first audio may be audio information in the first video; in this case, the process of obtaining the first audio may include:

[0132] Get the first video;

[0133] The audio information in the first video is obtained as the first audio.

[0134] Step 520, perform voice localization on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment containing voice.

[0135] The first audio may be composed of target audio segments; or the first audio may include target audio segments and non-target audio segments, and the non-target audio segments are audio segments that do not contain speech in the first audio. For example, when the first audio is a song, the first audio may include an audio segment corresponding to the lyrics and audio segments corresponding to the prelude, interlude and outro, wherein the audio segment corresponding to the lyrics is the target audio segment; the audio segment corresponding to the prelude, interlude and outro is the non-target audio segment. Alternatively, when the first audio is voice information in video content, the audio segment corresponding to the dialogue in the video content is the target audio segment; the audio segment corresponding to the ambient sound in the video content is the non-target audio segment.

[0136] Illustratively, the video content may include TV series, variety shows, movies, and UCG (User Generated Content) and other content displayed in the form of video.

[0137] Optionally, the process of performing voice localization on the first audio and obtaining at least one target audio segment in the first audio may be implemented as follows:

[0138] Performing voice positioning on the first audio to obtain time information of at least one target audio segment; the time information is used to indicate the start time and end time of the target audio segment;

[0139] At least one target audio segment is extracted from the first audio based on time information of the at least one target audio segment.

[0140] Among them, the time information may include the start and end time of the target audio segment; or, the time information may include the start time of the target audio segment and the duration of the target audio segment, or, the time information may also include the end time and duration of the target audio segment, thereby indicating the start time and end time of the target audio segment.

[0141] In the embodiment of the present application, a target audio segment may refer to an audio segment consisting of a group of continuous and uninterrupted speech frames in the first audio, wherein the speech frames are used to indicate audio frames of the speech type in the first audio.

[0142] Optionally, the computer device can determine that the two speech frames belong to different target audio segments when it is continuously detected that the number of frame intervals between two adjacent speech frames exceeds a frame number threshold; further, the computer device can obtain the speech frame that is earlier in time sequence as the end of the first target speech segment, and obtain the time information of the speech frame as the end time of the first target speech segment; obtain the speech frame that is later in time sequence as the beginning of the second target speech segment, and obtain the time information of the speech frame as the start time of the second target speech segment.

[0143] The process of performing voice localization on the first audio and obtaining time information of at least one target audio segment can be implemented as follows:

[0144] Extracting audio features from the first audio to obtain audio features of the first audio;

[0145] Based on the audio features of the first audio, voice positioning is performed on the first audio, and time information of at least one target audio segment is obtained.

[0146] The audio feature (Audio-embedding) of the first audio may be generated based on the spectrum information of the first audio, and the process may be implemented as follows:

[0147] Extracting frequency spectrum features of the first audio;

[0148] Based on the frequency spectrum characteristics of the first audio, generate an audio feature graph of the first audio;

[0149] Feature extraction is performed on the audio feature graph of the first audio to obtain audio features of the first audio.

[0150] The frequency spectrum feature of the first audio may be implemented as a Mel spectrum (Mel spectrum); wherein the Mel spectrum is a Mel filter performed on the basis of an audio STFT (Short-Time Fourier Transform); Figure 6 FIG. 4 shows a schematic diagram of audio feature extraction based on STFT provided by an exemplary embodiment of the present application, as shown in FIG. Figure 6 As shown, after acquiring the first audio 610, the computer device can perform frame processing 620 on the first audio according to the target length (time period or number of samples) to obtain multiple audio segments that are microscopically stable, and each audio segment is called a frame (CHUNK); after acquiring the multiple audio segments, in order to maintain the continuity of the audio signal, the computer device performs windowing processing 630 on the multiple audio segments; wherein the windowing processing refers to multiplying the audio signals corresponding to the multiple audio segments by a window function, so that the framed audio signals become continuous, and each frame of the audio signal will show the characteristics of a periodic function; Fourier transform is performed on each frame of the audio signal after the windowing processing 640 to obtain the STFT amplitude and STFT phase corresponding to each frame of the audio signal, so as to generate the following based on the STFT amplitude and STFT phase corresponding to each frame of the audio signal. Figure 7 An audio STFT energy spectrum 710 is shown.

[0151] The computer equipment performs Mel filtering on the obtained audio STFT energy spectrum; Mel frequency is proposed based on the auditory characteristics of the human ear, and Mel frequency has a nonlinear correspondence with frequency. Mel frequency cepstral coefficients (MFCC, Mel-FrequencyCepstral Coefficients) use the nonlinear correspondence between Mel frequency and frequency to calculate the Hz spectrum characteristics. In practical applications, the Mel filter is equivalent to filtering the frequency through a triangle band filter group; Figure 8 FIG. 4 shows a schematic diagram of a MEL filter bank shown in an exemplary embodiment of the present application, as shown in FIG. Figure 8 As shown, the computer device can define a filter group with M filters (the number of filters is close to the number of critical bands), and use the triangular band filter group to filter the audio STFT energy spectrum. The response of each filter at the center frequency is 1, and then decreases linearly until it reaches 0 at the center frequency of the adjacent triangular filter.

[0152] After Mel filtering is performed on the audio STFT energy spectrum, the logarithm of the result after Mel filtering is calculated, and finally a log-mel spectrum feature is generated, that is, the spectrum feature of the first audio is extracted; then, an audio feature map of the first audio is generated based on the log-mel spectrum feature; the audio feature map of the first audio is a two-dimensional map; then, based on the audio feature map of the first audio, the audio features of the first audio are obtained.

[0153] In an embodiment of the present application, the computer device may pre-process the first audio to obtain frequency spectrum features of the first audio, and generate an audio feature graph of the first audio based on the frequency spectrum features of the first audio.

[0154] Optionally, the computer device may process the audio feature graph of the first audio based on the audio localization network in the language recognition model to obtain the audio features of the first audio.

[0155] The audio localization network is a network obtained by training based on sample audio and audio segment labels of the sample audio; the audio localization network can be as follows Figure 4 The audio localization network in the language recognition model shown in the figure can be trained by referring to Figure 4 The relevant contents of the illustrated embodiment will not be repeated here.

[0156] The audio localization network may include a second feature extraction layer and an audio classification layer; the second feature extraction layer is used to process the audio feature map of the first audio to obtain the audio features of the first audio; the audio classification layer is used to obtain at least one target audio segment in the first audio based on the audio features of the first audio.

[0157] Optionally, the audio feature map is a feature map of a target size; the size of the audio feature map can be adapted to the requirements of the second feature extraction network; schematically, the second feature extraction layer can be implemented as a feature extraction network constructed based on a VGGish network, i.e., a VGGish-embedding network, where the VGGish network is a VGG (Visual Geometry Group) network based on Tensorflow, and the VGGish network supports the extraction of semantic 128-dimensional embedding feature vectors from audio waveforms; at this time, the audio feature map can be a 96*64 feature map formed by combining features of every 96 frames; since the number of frames of the first audio is greater than 96 frames, n audio feature maps of the first audio can be generated based on the spectral features of the first audio, where n is a positive integer; or, the second feature extraction layer can also be implemented as a feature extraction network constructed based on PANNS (Pretrained Audio Neural Networks), or a feature extraction network constructed based on other audio processing networks, and the like.

[0158] Taking the second feature extraction layer as the VGGish-embedding network as an example, Fig. 9 FIG. 4 shows a structural diagram of a VGGish-embedding network provided by an exemplary embodiment of the present application. Fig. 9 As shown, the input of the VGGish-embedding network is an audio feature sequence composed of n audio feature maps of the first audio, that is, n*96*64; the VGGish-embedding network is calculated based on the 96*64 basic feature map; Fig. 9It can be seen that the VGG network consists of 5 convolutional layers, 3 fully connected layers and a softmax output layer; each convolutional layer is separated by max-pooling; among them, the activation units of all hidden layers use the ReLU function. The VGG network uses multiple convolutional layers with smaller convolutional kernels (3*3) to replace a convolutional layer with a larger convolutional kernel. On the one hand, it can reduce parameters, and on the other hand, it is equivalent to performing more nonlinear mapping, which can increase the fitting and expression capabilities of the network. The size of the receptive field obtained by stacking two 3*3 convolutions is equivalent to a 5*5 convolution; and the receptive field obtained by stacking three 3*3 convolutions is equivalent to a 7*7 convolution, which can increase nonlinear mapping and reduce parameters. After being processed by the VGGish network, each 96*64 audio feature map will be calculated into a 128-order embedding vector 910. This 128-order embedding vector 910 contains all the information of 96 frames of audio signals, which is the high-level features output by the VGGish network. The VGGish-embedding network may eventually generate an n*128 embedding vector sequence as the audio feature of the first audio.

[0159] When voice positioning of the first audio is performed based on the audio features of the first audio to obtain time information of at least one target audio segment, the audio features of the first audio can be processed based on the audio classification layer in the audio positioning network to obtain time information of at least one target audio segment.

[0160] Among them, the audio classification layer can be composed of a CNN (Convolutional Neural Network), a BLSTM (Bidirectional Long Short-term Memory) network and a Dense (dense connection) network. Fig.10 FIG. 4 shows a schematic diagram of the structure of an audio classification layer shown in an exemplary embodiment of the present application. Fig.10As shown, since the embedding vector is a 128-order vector, and different audio events have different characteristic performances in different frequency bands, when processing the 128-order vector, it is necessary to first integrate the high- and low-frequency information; that is, when the upstream network (the second feature extraction layer) inputs the generated n 128-order embedding vectors into the audio classification layer 1010 respectively, the computer device uses several convolutional layers in CNN1011 to integrate the information in the high- and low-frequency dimensions; the features processed by CNN1011 are input into the bidirectional LSTM network (BLSTM) 1012 to integrate the long- and short-time information; the features processed by the bidirectional LSTM network are input into the fully connected layer for speech binary classification, and the speech posterior probability of the embedding vector is calculated. In the embodiment of the present application, the fully connected layer can be implemented as follows Fig.10 The densely connected network (Dense network) 1013 shown, or the fully connected layer can also be implemented as other connection networks, which is not limited in the present application; repeat the above process until the speech posterior probability of the audio feature sequence composed of n embedding vectors is obtained, that is, the posterior probability sequence of speech recognition positioning, and the speech posterior probability is used to indicate the probability that the audio frame is a speech frame; the posterior probability sequence of speech recognition positioning is regularized, and the audio frames are screened based on the set probability threshold to locate at least one target audio segment containing speech in the first audio.

[0161] Step 530: Process at least one target audio segment respectively based on the first feature extraction layer in the language recognition network to obtain audio features of at least one target audio segment.

[0162] After locating at least one target audio segment in the first audio through the audio localization network in the language recognition model, the computer device can segment the first audio based on time information of each target audio segment to obtain at least one target audio segment.

[0163] The language recognition network is a network obtained by training based on sample audio segments and language labels of the sample audio segments; the language recognition network can be as follows: Figure 4 The language recognition network in the language recognition model shown in the figure can be trained by referring to Figure 4 The relevant contents in the illustrated embodiment will not be repeated here.

[0164] When obtaining the audio features of at least one target audio segment, the computer device can divide the at least one target audio segment into a frame sequence consisting of m audio frames, and then process the frame sequence corresponding to each target audio segment based on the first feature extraction layer in the language recognition network to obtain the audio features of each target audio frame segment.

[0165] Optionally, the first feature extraction layer can be implemented as a VGGish network or a PANNS network. Taking the first feature extraction layer implemented as a VGGish network as an example, Fig.11 A schematic diagram of an audio feature extraction process provided by an exemplary embodiment of the present application is shown, taking a first audio segment as an example, the first audio segment is any audio segment in at least one target audio segment. Fig.11 As shown, after obtaining the first audio paragraph 1110, the first audio paragraph 1110 is divided into m audio frames to obtain a frame sequence of the first audio paragraph; the frame sequence of the first audio paragraph is input into the VGGish network 1120; the VGGish network 1120 processes the m audio frames in the frame sequence respectively to obtain audio features corresponding to each of the m audio frames, and obtains an audio feature sequence composed of the audio features corresponding to each of the m audio frames as the audio features of the first audio paragraph.

[0166] In the embodiment of the present application, the audio features of a target audio segment can be implemented as an m*128 two-dimensional matrix.

[0167] Step 540: Process the audio features of at least one target audio segment based on the language recognition layer in the language recognition network to obtain the language type of the at least one target audio segment.

[0168] Schematically, the language recognition layer can be composed of a BLSTM network and a fully connected layer; Fig.12 FIG. 4 shows a schematic diagram of the structure of a language recognition layer shown in an exemplary embodiment of the present application, taking the first audio segment as an example; Fig.12As shown, the computer device inputs the audio features of the first audio segment into a two-layer bidirectional LSTM network (BLSTM network) 1210, and the number of hidden layer nodes of each LSTM layer can be 512; extract the forward hidden state and reverse hidden state of the last layer, and the forward hidden state and reverse hidden state are both a 1*512 vector; concatenate the two vectors to generate a 1024 information vector 1220, which contains the language information of the first audio segment. Finally, the 1024 information vector is input into the fully connected layer 1230 to generate the confidence of each language type; wherein the activation function in the fully connected layer 1230 is a softmax function, and the fully connected layer can be implemented as an MLP (Multilayer Perceptron) neural network; the computer device can determine the language type with the highest confidence as the language type of the first audio segment. Based on the above process, at least one target audio segment is processed separately to obtain the language type of at least one target audio segment.

[0169] Step 550: Acquire the language type of the first audio based on the language type of at least one target audio segment.

[0170] In a possible implementation, the process of obtaining the language type of the first audio may be implemented as follows:

[0171] Based on the language type of at least one target audio segment, obtaining the cumulative duration of the target audio segments corresponding to each language type;

[0172] Based on the accumulated durations corresponding to the respective language types, the language type of the first audio is obtained.

[0173] Optionally, the computer device may count the cumulative duration of the target audio segments corresponding to the language type involved in the first audio in each language type; illustratively, if among 10 language types, only 3 of the language types are involved in the first audio, then when performing statistics, the cumulative duration of the target audio segments contained in each of the 3 language types is counted to obtain the language type of the first audio from the 3 language types, so as to reduce the waste of computing resources of the computer device.

[0174] Optionally, the computer device may determine the language type with the longest cumulative duration among the language types as the language type of the first audio;

[0175] or,

[0176] The computer device may determine, among the various language types, a language type having the largest ratio between the cumulative duration and the duration of the first audio as the language type of the first audio.

[0177] Furthermore, the computer device may also determine the language type having the largest ratio between the accumulated duration and the total duration of at least one target audio segment as the language type of the first audio.

[0178] That is to say, the computer device determines the language type that exists for the longest time in the first audio as the language type of the first audio.

[0179] In one possible scenario, not all audio content in the first audio is valid content, that is, the first audio contains valid content and invalid content; wherein, the valid content may refer to the audio content in the first audio uploaded by the uploader; the invalid content may refer to other content in the first audio corresponding to the audio content uploaded by the uploader in addition to the audio content uploaded by the uploader; illustratively, the audio content uploaded by the uploader is 5 minutes long, and the first audio, in addition to the audio content, also contains an additional 2 minutes of advertising content. At this time, the first audio contains 5 minutes of valid content and 2 minutes of invalid content.

[0180] In this case, when obtaining the cumulative duration of the target audio segments corresponding to each language type, in order to reduce the influence of the invalid content in the first audio on the language type identification of the first audio, based on the language type of at least one target audio segment, the process of obtaining the cumulative duration of the target audio segments corresponding to each language type can be implemented as follows:

[0181] Obtaining position information of at least one target audio segment in the first audio; the position information is used to indicate the time position of the target audio segment in the first audio;

[0182] Based on the position information of the at least one target audio segment in the first audio, weighted processing is performed on the segment duration of the at least one target audio segment;

[0183] Based on the language type of at least one target audio segment and the weighted segment duration of at least one target audio segment, the cumulative durations of the target audio segments corresponding to the respective language types are obtained.

[0184] Considering that invalid content in audio generally appears at the beginning or end of the audio, an audio interval may be set in the computer device, and the audio interval may be set based on the ratio of the duration to the duration of the first audio; for example, the first 10% of the audio interval of the first audio is set as the first audio interval, and the last 10% of the audio interval after the first audio is set as the second audio interval; the middle 80% of the audio interval of the first audio is set as the third audio interval; different audio intervals may correspond to different weight coefficients, and the weight coefficients corresponding to every two audio intervals may be the same or different; wherein the weight coefficient corresponding to the third audio interval is greater than the weight coefficient corresponding to the first audio interval, and the weight coefficient corresponding to the third audio interval is greater than the weight coefficient corresponding to the second audio interval; schematically, for a 5-minute first audio, the weight coefficient corresponding to the first 30 seconds of audio (first audio interval) is 0.5, the weight coefficient corresponding to the last 30 seconds of audio (second audio interval) is 0.5, and the weight coefficient corresponding to the middle 4 minutes (third audio interval) is 1.

[0185] After obtaining the position information of at least one target audio segment in the first audio, the weight coefficient corresponding to each target audio segment can be determined based on the audio interval in which each target audio segment is located to obtain the segment duration of each target audio segment; and then the cumulative duration of the target audio segments corresponding to each language type is counted.

[0186] In another possible implementation, the process of obtaining the position information of at least one target audio segment in the first audio may also be implemented as follows:

[0187] Obtaining a reference time point of at least one target audio segment in the first audio; the reference time point is one of a start time point, a middle time point, and an end time point of the audio segment;

[0188] Based on the reference time points of the at least one target audio segment in the first audio, respectively, obtaining the offset duration of the at least one target audio segment as the position information of the at least one target audio segment in the first audio; the offset duration is the duration between the reference time point and the middle time point of the first audio;

[0189] In the above case, based on the position information of the at least one target audio segment in the first audio, weighted processing is performed on the segment duration of the at least one target audio segment, including:

[0190] Obtaining a weight of at least one target audio segment; the weight is inversely correlated with the offset duration;

[0191] Based on the weight of the at least one target audio segment, a weighted processing is performed on the segment duration of the at least one target audio segment.

[0192] That is to say, when obtaining the weight of at least one target audio segment, the weight of each target audio segment can be obtained based on the offset of the reference time point of each target audio segment relative to the middle time point of the first audio; the farther away from the middle time point of the first audio, the smaller the weight, and the closer to the middle time point of the first audio, the larger the weight; wherein, the inverse correlation between the weight and the offset duration can be set based on actual needs, and this application does not impose any restrictions on this.

[0193] In another possible implementation, when the computer device acquires the language type of the first audio based on the language type of each of the at least one target audio segments, it may also count the number of audio segments of each language type in the at least one target audio segment based on the language type of each of the at least one target audio segments;

[0194] Among the language types, a language type that contains the largest number of target audio segments is determined as the language type of the first audio.

[0195] That is to say, the computer device may determine the language type that appears most frequently in the first audio as the language type of the first audio.

[0196] In summary, the audio processing method provided in the embodiment of the present application extracts the target audio segments containing speech in the first audio to obtain the audio features of each target audio segment, so as to obtain the language type of each target audio segment based on the audio features of each target audio segment, and then obtains the language type of the first audio based on the language type of each target audio segment; in the above scheme, the language of the audio is automatically acquired by performing language recognition on each audio segment containing speech in the audio, thereby avoiding the manual labeling process in the process of acquiring the language type of the first audio, simplifying the language recognition process, improving the efficiency of acquiring the language type of audio and video content, and improving the accuracy of acquiring the language type.

[0197] For example, the first audio is the audio information in the first video. Fig.13 A schematic diagram of an audio processing process shown in an exemplary embodiment of the present application is shown. Fig.13As shown, after obtaining the first video 1310, the first audio 1320 is obtained from the first video 1310, and the first audio 1320 is input into the audio localization network 1331 of the language recognition model 1330. Based on the time information of each target audio segment output by the audio localization network 1331, the first audio is segmented to obtain at least one target audio segment 1332; at least one target audio segment is input into the language recognition network 1333 in the language recognition model 1330, and the audio features 1334 of each target audio segment are obtained through the first feature extraction layer in the language recognition network 1333; the audio features 1334 of each target audio segment are input into the language recognition layer to obtain the language type 1335 corresponding to each target audio segment; based on the language type corresponding to each target audio segment, the language type of the first audio or the first video is obtained. Thus, the recommendation side can classify and recommend the first video based on the language type of the first video.

[0198] Fig.14 FIG. 1 shows an application diagram of an audio processing method provided by an exemplary embodiment of the present application. Fig.14 As shown, a video application with a video recommendation function or a video playback function may include a large number of video contents, which may be video resource contents or UCG contents. For video contents including spoken videos 1410, that is, video contents including voice clips, the video application may process the audio information in each spoken video by the audio processing method provided in the present application to obtain the language type 1420 of each spoken video, thereby classifying each spoken video based on the language type or annotating the language label, so that when the spoken video is recommended on the recommendation side, the video recommendation may be based on the language type of the spoken video.

[0199] Fig.15 A block diagram of an audio processing device exemplarily shown in the present application is shown in FIG. Fig.15 As shown, the audio processing device may include:

[0200] The audio acquisition module 1510 is used to acquire the first audio;

[0201] The speech localization module 1520 is used to perform speech localization on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment containing speech;

[0202] A feature acquisition module 1530, configured to acquire audio features of at least one of the target audio segments;

[0203] A language identification module 1540 is used to perform language identification based on the audio features of at least one of the target audio segments to obtain the language type of at least one of the target audio segments;

[0204] The language acquisition module 1550 is used to acquire the language type of the first audio based on the language type of at least one of the target audio segments.

[0205] In a possible implementation, the feature acquisition module 1530 is used to process at least one of the target audio segments based on the first feature extraction layer in the language recognition network to obtain audio features of at least one of the target audio segments;

[0206] The language recognition module 1540 is used to process the audio features of at least one target audio segment based on the language recognition layer in the language recognition network to obtain the language type of at least one target audio segment;

[0207] The language recognition network is a network obtained by training based on sample audio segments and language labels of the sample audio segments.

[0208] In a possible implementation, the voice localization module 1520 includes:

[0209] A voice positioning submodule, configured to perform voice positioning on the first audio and obtain time information of at least one of the target audio segments; the time information is used to indicate the start time and end time of the target audio segment;

[0210] The segment extraction submodule is used to extract at least one target audio segment from the first audio based on the time information of at least one target audio segment.

[0211] In a possible implementation, the voice localization submodule includes:

[0212] a feature acquisition unit, configured to extract audio features from the first audio to obtain audio features of the first audio;

[0213] A time information acquisition unit is used to perform voice localization on the first audio based on the audio features of the first audio, and acquire the time information of at least one of the target audio segments.

[0214] In a possible implementation, the feature acquisition unit is used to:

[0215] Extracting frequency spectrum features of the first audio;

[0216] Based on the frequency spectrum characteristics of the first audio, generating an audio feature graph of the first audio;

[0217] Feature extraction is performed on the audio feature graph of the first audio to obtain audio features of the first audio.

[0218] In a possible implementation, the feature acquisition unit is used to:

[0219] Processing the audio feature graph of the first audio based on an audio localization network to obtain audio features of the first audio;

[0220] The time information acquisition unit is used to process the audio features of the first audio based on the audio classification layer in the audio localization network to obtain the time information of at least one target audio segment;

[0221] The audio localization network is a network obtained by training based on sample audio and audio segment labels of the sample audio.

[0222] In a possible implementation, the language acquisition module 1550 is used to count the number of audio segments belonging to each language type in at least one of the target audio segments based on the language type of each of the at least one target audio segments;

[0223] The language type containing the largest number of the target audio segments among the various language types is determined as the language type of the first audio.

[0224] In a possible implementation, the language acquisition module 1550 includes:

[0225] A cumulative duration acquisition submodule, configured to acquire the cumulative durations of the target audio segments corresponding to the respective language types based on the language type of at least one of the target audio segments;

[0226] The language acquisition submodule is used to acquire the language type of the first audio based on the accumulated durations corresponding to each language type.

[0227] In a possible implementation, the language acquisition submodule is used to:

[0228] Determine the language type with the longest cumulative duration among the various language types as the language type of the first audio;

[0229] or,

[0230] Among the language types, the language type having the largest ratio between the accumulated duration and the duration of the first audio is determined as the language type of the first audio.

[0231] In a possible implementation, the accumulated duration acquisition submodule includes:

[0232] a position information acquisition unit, configured to acquire position information of at least one of the target audio segments in the first audio; the position information is used to indicate a time position of the target audio segment in the first audio;

[0233] a weighted processing unit, configured to perform weighted processing on the duration of each of at least one of the target audio segments based on the position information of at least one of the target audio segments in the first audio;

[0234] The cumulative duration acquisition unit is used to acquire the cumulative durations of the target audio segments corresponding to each language type based on the language type of at least one of the target audio segments and the weighted duration of at least one of the target audio segments.

[0235] In a possible implementation, the location information acquiring unit is used to:

[0236] Obtaining a reference time point of at least one of the target audio segments in the first audio; the reference time point is one of a starting time point, a middle time point, and an ending time point of the audio segment;

[0237] Based on a reference time point of at least one of the target audio segments in the first audio, respectively, obtaining an offset duration of at least one of the target audio segments as position information of at least one of the target audio segments in the first audio; the offset duration is a duration between the reference time point and an intermediate time point of the first audio;

[0238] The weighted processing unit is used to obtain a weight of at least one of the target audio segments; the weight is inversely correlated with the offset duration;

[0239] Based on the weight of at least one of the target audio segments, weighted processing is performed on the segment duration of at least one of the target audio segments.

[0240] In summary, the audio processing device provided in the embodiment of the present application extracts the target audio segments containing speech in the first audio to obtain the audio features of each target audio segment, so as to obtain the language type of each target audio segment based on the audio features of each target audio segment, and then obtains the language type of the first audio based on the language type of each target audio segment; in the above scheme, the language of the audio is automatically acquired by performing language recognition on each audio segment containing speech in the audio, thereby avoiding the manual labeling process in the process of acquiring the language type of the first audio, simplifying the language recognition process, improving the efficiency of acquiring the language type of audio and video content, and improving the accuracy of acquiring the language type.

[0241] Fig.16 The block diagram of the computer device 1600 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as each server in the above-mentioned scheme of the present application. The computer device 1600 includes a processor (such as a central processing unit (CPU)) 1601, a system memory 1604 including a random access memory (RAM) 1602 and a read-only memory (ROM) 1603, and a system bus 1605 connecting the system memory 1604 and the processor 1601. The computer device 1600 also includes a mass storage device 1606 for storing an operating system 1609, an application program 1610 and other program modules 1611.

[0242] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1604 and mass storage device 1606 can be collectively referred to as memory.

[0243] According to various embodiments of the present application, the computer device 1600 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1600 can be connected to the network through the network interface unit 1607 connected to the system bus 1605, or the network interface unit 1607 can be used to connect to other types of networks or remote computer systems (not shown).

[0244] The memory further stores at least one computer program, and the processor 1601 implements all or part of the steps in the audio processing method shown in the above embodiments by executing the at least one computer program.

[0245] Fig.17 FIG. 1 is a block diagram of a computer device 1700 according to an exemplary embodiment of the present application. The computer device 1700 may be implemented as the terminal described above.

[0246] Typically, the computer device 1700 includes a processor 1701 and a memory 1702 .

[0247] Processor 1701 may include one or more processing cores, such as a 4-core processor, a 17-core processor, etc. Processor 1701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, processor 1701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, processor 1701 may also include an AI processor, which is used to process computing operations related to machine learning.

[0248] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1702 is used to store at least one instruction, which is used to be executed by the processor 1701 to implement all or part of the steps in the audio processing method shown in the method embodiment of the present application.

[0249] In some embodiments, the computer device 1700 may also optionally include: a peripheral device interface 1703 and at least one peripheral device. The processor 1701, the memory 1702 and the peripheral device interface 1703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707 and a power supply 1708.

[0250] In some embodiments, the computer device 1700 further includes one or more sensors 1709 , including but not limited to: an acceleration sensor 1710 , a gyroscope sensor 1711 , a pressure sensor 1712 , an optical sensor 1713 , and a proximity sensor 1714 .

[0251] Those skilled in the art will understand that Fig.17The structure shown in the figure does not constitute a limitation on the computer device 1700, and the computer device 1700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0252] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement all or part of the steps in the above audio processing method. For example, the computer-readable storage medium can be a read-only memory, a random access memory, a read-only optical disk, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0253] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor Figure 2 , Figure 4 or Figure 5 All or part of the steps of the audio processing method shown in any embodiment shown.

[0254] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0255] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that: The method comprises: Get the first audio; Performing voice localization on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment containing voice; Acquire audio features of at least one of the target audio segments; Performing language recognition based on the audio features of at least one of the target audio segments to obtain the language type of at least one of the target audio segments; Acquire position information of at least one of the target audio segments in the first audio; the position information is used to indicate the time position of the target audio segment in the first audio; Based on the position information of at least one target audio segment in the first audio, weighting the segment duration of at least one target audio segment; Based on the language type of at least one of the target audio segments and the weighted segment duration of at least one of the target audio segments, obtaining the cumulative duration of the target audio segments corresponding to each language type; Based on the accumulated durations corresponding to the respective language types, the language type of the first audio is acquired.

2. The method according to claim 1, characterized in that The obtaining of the audio features of at least one of the target audio segments includes: Processing at least one of the target audio segments based on the first feature extraction layer in the language recognition network to obtain audio features of at least one of the target audio segments; The performing language recognition based on the audio features of at least one of the target audio segments to obtain the language type of at least one of the target audio segments includes: Based on the language recognition layer in the language recognition network, the audio features of at least one of the target audio segments are processed respectively to obtain the language type of the at least one target audio segment; The language recognition network is a network obtained by training based on sample audio segments and language labels of the sample audio segments.

3. The method according to claim 1, characterized in that The performing voice localization on the first audio to obtain at least one target audio segment in the first audio includes: Performing voice positioning on the first audio to obtain time information of at least one of the target audio segments; the time information is used to indicate the start time and end time of the target audio segment; At least one target audio segment is extracted from the first audio based on the time information of at least one target audio segment.

4. The method according to claim 3, characterized in that The performing voice localization on the first audio to obtain time information of at least one target audio segment includes: Performing audio feature extraction on the first audio to obtain audio features of the first audio; The first audio is voice-localized based on the audio features of the first audio, and the time information of at least one of the target audio segments is acquired.

5. The method according to claim 4, characterized in that The extracting audio features from the first audio to obtain audio features of the first audio includes: Extracting frequency spectrum features of the first audio; Based on the frequency spectrum characteristics of the first audio, generating an audio feature graph of the first audio; Feature extraction is performed on the audio feature graph of the first audio to obtain audio features of the first audio.

6. The method according to claim 5, characterized in that The extracting features from the audio feature graph of the first audio to obtain the audio features of the first audio includes: Processing the audio feature graph of the first audio based on an audio localization network to obtain audio features of the first audio; The performing voice localization on the first audio based on the audio feature of the first audio to obtain the time information of at least one of the target audio segments includes: Based on the audio classification layer in the audio localization network, the audio features of the first audio are processed to obtain the time information of at least one of the target audio segments; The audio localization network is a network obtained by training based on sample audio and audio segment labels of the sample audio.

7. The method according to claim 1, characterized in that The acquiring the language type of the first audio based on the accumulated durations respectively corresponding to the language types includes: Determine the language type with the longest cumulative duration among the various language types as the language type of the first audio; or, Among the language types, the language type having the largest ratio between the accumulated duration and the duration of the first audio is determined as the language type of the first audio.

8. The method according to claim 1, characterized in that The obtaining of position information of at least one target audio segment in the first audio includes: Obtaining a reference time point of at least one of the target audio segments in the first audio; the reference time point is one of a starting time point, a middle time point, and an ending time point of the audio segment; Based on a reference time point of at least one of the target audio segments in the first audio, respectively, obtaining an offset duration of at least one of the target audio segments as position information of at least one of the target audio segments in the first audio; the offset duration is a duration between the reference time point and an intermediate time point of the first audio; The weighting process of the duration of each of the at least one target audio segment based on the position information of the at least one target audio segment in the first audio includes: Obtaining a weight of at least one of the target audio segments; the weight is inversely correlated with the offset duration; Based on the weight of at least one of the target audio segments, weighted processing is performed on the segment duration of at least one of the target audio segments.

9. An audio processing device, characterized in that: The device comprises: An audio acquisition module, used to acquire a first audio; A speech localization module, configured to perform speech localization on the first audio to obtain at least one target audio segment in the first audio; the target audio segment is an audio segment containing speech; A feature acquisition module, used to acquire audio features of at least one of the target audio segments; A language recognition module, configured to perform language recognition based on the audio features of at least one of the target audio segments, and obtain the language type of each of the at least one of the target audio segments; A language acquisition module, configured to acquire the language type of the first audio based on the language type of at least one of the target audio segments; Wherein, the language acquisition module includes: A cumulative duration acquisition submodule, configured to acquire the cumulative durations of the target audio segments corresponding to the respective language types based on the language type of at least one of the target audio segments; A language acquisition submodule, used for acquiring the language type of the first audio based on the accumulated durations corresponding to the respective language types; The cumulative duration acquisition submodule includes: a position information acquisition unit, configured to acquire position information of at least one of the target audio segments in the first audio; the position information is used to indicate a time position of the target audio segment in the first audio; a weighted processing unit, configured to perform weighted processing on the duration of each of at least one of the target audio segments based on the position information of at least one of the target audio segments in the first audio; The cumulative duration acquisition unit is used to acquire the cumulative durations of the target audio segments corresponding to each language type based on the language type of at least one of the target audio segments and the weighted duration of at least one of the target audio segments.

10. A computer device, characterized in that: The computer device comprises a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the audio processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the audio processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises at least one computer program, and the computer program is loaded and executed by a processor to implement the audio processing method according to any one of claims 1 to 8.

Images