Audio recognition method, apparatus, device, and storage medium

Abstract

This disclosure provides an audio recognition method, apparatus, device, and storage medium, applied in the field of audio detection and analysis technology. The method includes inputting acquired audio to be recognized into a silence detection model, outputting multiple audio segments; determining silence segments based on a window detection algorithm using consecutive frames, according to the audio segments; extracting audio segments voc1 and voc2 from the silence segments, inputting them into a voiceprint feature extraction model, outputting voiceprint features fea1 for audio segment Voc1 and fea2 for audio segment voc2; calculating the sim1_2 values ​​of voiceprint features fea1 and fea2 based on cosine similarity compute_cosine_socre; determining speaker information based on voiceprint features fea1 and a preset voiceprint database in response to a sim1_2 value greater than a preset sim_threshold value; and inputting audio segments Voc1 and voc2 into a speech content recognition model, outputting the speech content. In this manner, more accurate speaker recognition and speech content recognition can be obtained, effectively completing the recording and generation of the audio to be recognized.

Description

Technical Field

[0001] This disclosure relates to the field of audio detection and analysis technology, and in particular to audio recognition methods, devices, equipment, and storage media. Background Technology

[0002] With the increasing importance of meeting audio file recognition, accurately identifying speakers and their content, and effectively recording meeting audio, has become a pressing issue in today's society. Meeting audio recognition requires identifying speakers, speaking time periods, and speaking content. This typically necessitates effective segmentation of the input audio. Traditional methods using silence detection models often result in fragmented audio segmentation, hindering speaker identification and causing subsequent speech recognition to lose contextual relevance, leading to cumulative errors in speaker and content recognition. Another approach involves extracting acoustic features from each frame of the audio to be segmented, performing semantic boundary sequence annotation, and segmenting the audio based on these semantic boundary results. However, this also faces challenges in frame-level audio annotation, leading to biases in the trained model and difficulties in boundary recognition. Yet another method involves removing silence segments from the audio to be segmented to reduce their impact on final speech separation, and using endpoint detection to separate speech segments from different speakers within the same audio segment, avoiding the inclusion of multiple speakers in the same audio segment. However, this approach also suffers from inaccurate endpoint detection.

[0003] Therefore, there is an urgent need for an audio recognition method that can more accurately identify the speaker and the content of the speech. Summary of the Invention

[0004] This disclosure provides an audio recognition method, apparatus, device, and storage medium.

[0005] According to a first aspect of this disclosure, an audio recognition method is provided. The method includes:

[0006] Acquire the audio to be identified; input the audio to be identified into the silence detection model, and output multiple audio segments;

[0007] Based on the audio segment, a window detection algorithm based on consecutive frames is used to determine the silent segment;

[0008] Based on the silent segment, extract audio segments voc1 and voc2, input them into the voiceprint feature extraction model, and output the voiceprint feature fea1 of the audio segment voc1 and the voiceprint feature fea2 of the audio segment voc2.

[0009] The sim1_2 values ​​of the voiceprint features fea1 and fea2 are calculated based on the cosine similarity compute_cosine_socre.

[0010] In response to the sim1_2 value being greater than the preset sim_threshold value, the speaker information is determined based on the voiceprint feature fea1 and the preset voiceprint database; and the audio segment voc1 and the audio segment voc2 are input into the speech content recognition model to output the speech content.

[0011] As described above and in any possible implementation, a further implementation is provided, wherein the audio to be identified is input into the silence detection model, and multiple audio segments are output, including:

[0012] The VAD model is used to detect silent endpoints and identify silent states in the audio to be identified.

[0013] Based on the endpoint detection results, the audio to be identified is marked with boundaries to obtain multiple audio segments.

[0014] In addition to the aspects and any possible implementations described above, an implementation is further provided in which the audio segment is an audio fragment of consecutive frames;

[0015] The step of determining the silent segment based on the audio segment and a window detection algorithm for consecutive frames includes:

[0016] The continuous frame-based window detection algorithm counts the number of frames in the audio segments marked as silent, and determines the silent segments based on the number of continuous frames.

[0017] In addition to the aspects described above and any possible implementation, a further implementation is provided, wherein the extraction of audio segments voc1 and voc2 based on the silence segment includes: extracting the audio segment preceding the silence segment as audio segment voc1, and extracting the audio segment following the silence segment as audio segment voc2.

[0018] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the preset voiceprint database includes multiple speaker information and their corresponding voiceprint features;

[0019] The step of determining speaker information based on the voiceprint feature fea1 and a preset voiceprint database includes:

[0020] The voiceprint feature fea1 and the sim1_2 value of each voiceprint feature in the voiceprint database are calculated based on the cosine similarity compute_cosine_socre.

[0021] If the largest sim1_2 value is greater than the sim_threshold value, then the speaker information corresponding to the voiceprint feature corresponding to the largest sim1_2 value is used as the speaker information corresponding to the voiceprint feature fea1.

[0022] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the method further includes:

[0023] In response to the sim1_2 value being less than or equal to the sim_threshold value, the corresponding speaker information is determined based on the voiceprint feature fea1 and a preset voiceprint database; and the audio segment voc1 is input into the speech content recognition model to output the corresponding speech content.

[0024] The audio segment voc2 is further processed for audio recognition until all audio segments are recognized, and the corresponding speaker information and speech content are output.

[0025] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the audio segment includes timestamp information;

[0026] The method further includes:

[0027] Based on the timestamp information, corresponding start and end times are generated for the output speaker information and speech content.

[0028] According to a second aspect of this disclosure, an audio recognition device is provided. The device includes:

[0029] The audio acquisition and recognition module is used to acquire the audio to be recognized; input the audio to be recognized into the silence detection model, and output multiple audio segments;

[0030] A silence determination module is used to determine a silence segment based on the audio segment and a window detection algorithm based on consecutive frames;

[0031] The voiceprint extraction module is used to extract audio segments voc1 and voc2 based on the silent segment, input the voiceprint feature extraction model, and output the voiceprint feature fea1 of the audio segment voc1 and the voiceprint feature fea2 of the audio segment voc2.

[0032] The voiceprint calculation module is used to calculate the sim1_2 values ​​of the voiceprint features fea1 and fea2 based on the cosine similarity compute_cosine_socre.

[0033] The content recognition module is used to determine the speaker information based on the voiceprint feature fea1 and the preset voiceprint database in response to the sim1_2 value being greater than the preset sim_threshold value; and to input the audio segment voc1 and the audio segment voc2 into the speech content recognition model and output the speech content.

[0034] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0035] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method as described in the first aspect of this disclosure.

[0036] The audio recognition method, apparatus, device, and storage medium provided in the embodiments of this disclosure, for a given input audio to be recognized, employ a silence detection model and a voiceprint feature extraction model to perform fusion judgment on audio segmentation. The audio of the same person will not be segmented due to silence, and the resulting fusion segmentation will make the audio segmentation more accurate. Performing voiceprint recognition and speech content recognition on the segmented audio can obtain more accurate speaker recognition and speech content recognition, effectively completing the recording and generation of the audio to be recognized.

[0037] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0038] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0039] Figure 1 A flowchart of an audio recognition method according to an embodiment of the present disclosure is shown;

[0040] Figure 2 A block diagram of an audio recognition device according to an embodiment of the present disclosure is shown;

[0041] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0043] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0044] In this disclosure, the input audio to be identified (e.g., a conference audio file) is segmented using a silence determination VAD model and a voiceprint extraction model. This avoids the problem of overly fragmented segmentation that relies entirely on silence determination, resulting in segmented audio. Then, voiceprint features and prosodic features are used to perform speaker voiceprint recognition and speech content recognition on the segmented audio, obtaining speaker information and speech content information, and achieving the optimal speaker recognition and speech content recognition results.

[0045] Figure 1 A flowchart of an audio recognition method 100 according to an embodiment of the present disclosure is shown. Method 100 includes:

[0046] Step 110: Obtain the audio to be identified; input the audio to be identified into the silence detection model and output multiple audio segments.

[0047] In some embodiments, when it is necessary to automatically identify and output the corresponding text record and time information for a certain audio segment, such as a meeting audio file, it is necessary to obtain the audio to be identified.

[0048] In some embodiments, the VAD model is used to detect silence endpoints and mark silences in the audio to be identified; based on the endpoint detection results, the audio to be identified is marked with boundaries to obtain multiple audio segments.

[0049] Specifically, the silence detection model is an improved VAD model to improve the accuracy of silence endpoint detection and recognition in the audio to be identified. The VAD network employs a convolutional layer + transformer structure, with a sampling rate of 16k, a thread count of 1, a model threshold of 0.5, a minimum silence duration of 0.5, a minimum speech duration of 0.25, a window size of 512 frames, a buffer size in seconds of 60 seconds, an audio feature dimension of 80, and the minimum trailing silence rule1_min_trailing_silence for rule 1 set to 2.4, the minimum trailing silence rule2_min_trailing_silence for rule 2 set to 1.2, and the minimum utterance length rule3_min_utterance_length for rule 3 set to 300. The audio to be identified, obtained in step 110, is input into the silence detection model. This allows for the detection and marking of silence endpoints in the audio. Then, based on the endpoint detection results, boundary marking is applied to the audio to be identified, thus segmenting it and ultimately forming an audio sequence of multiple segments marked with silence indicators, i.e., audio segments. For example, after boundary marking, the audio to be identified is divided into segments a, b, and c, where segment b is marked as silent.

[0050] Step 120: Based on the audio segment, determine the silent segment using a window detection algorithm based on consecutive frames.

[0051] In some embodiments, the audio segment is an audio fragment of consecutive frames; the step of determining the silent segment based on the audio segment and a window detection algorithm based on consecutive frames includes: using a window detection algorithm based on consecutive frames to count the number of frames of the audio segments marked as silent, and verifying that the audio segments with a number of consecutive frames exceeding a preset threshold (e.g., 8) are silent and do not require adjustment of the silence marking result, wherein the first audio segment marked as silent is taken as the silent segment.

[0052] In some embodiments, the silent segment in the above embodiments can also be all audio segments marked as silent in the audio to be identified. Then, determining the preceding and following audio segments based on the silent segments and whether they belong to the same speaker can be done simultaneously to improve recognition efficiency. This method is especially efficient when there are three or more different speakers in the audio to be identified.

[0053] In some embodiments, to further accurately determine whether the audio segments identified as silent in step 110 are truly silent, a predetermined threshold needs to be set to ultimately confirm them as silent segments. This involves using a continuous frame window detection algorithm. A segment is considered silent only when n consecutive frames (e.g., n = 10) are all silent, meaning the number of consecutive frames exceeds a preset threshold. This eliminates instances where short pauses in the same person's speech cause the audio segments to be identified as silent, allowing for better sample selection for subsequent voiceprint identification and improving detection efficiency. It also further refines the output of the silence detection model.

[0054] Step 130: Extract audio segments voc1 and voc2 based on the silence segment, input the voiceprint feature extraction model, and output the voiceprint feature fea1 of the audio segment voc1 and the voiceprint feature fea2 of the audio segment voc2.

[0055] In some embodiments, after determining the silence segment in step 120, it is necessary to determine whether the audio segments at the two adjacent ends of the silence segment belong to the same person. At this point, it is necessary to extract audio segments voc1 and voc2 based on the silence segment, including: extracting the audio segment preceding the silence segment as audio segment voc1, and extracting the audio segment following the silence segment as audio segment voc2. Audio segments voc1 and voc2 are then input into a voiceprint feature extraction model, outputting the voiceprint feature fea1 of audio segment voc1 and the voiceprint feature fea2 of audio segment voc2. The voiceprint feature extraction model uses a 34-layer ResNet network to extract 512-dimensional voiceprint features, with a resampling rate of 16k, Mel-spectral cepstral coefficients of 80, a frame length of 25 frames, a frame shift of 10 frames, and a feature extraction parameter of SamplesPerChunk of 80k (5 seconds).

[0056] Step 140: Calculate the sim1_2 values ​​of the voiceprint features fea1 and fea2 based on the cosine similarity compute_cosine_socre.

[0057] In some embodiments, similarity is calculated using the voiceprint features fea1 of audio segment voc1 and fea2 of audio segment voc2 obtained in step 130. Specifically, cosine similarity compute_cosine_socre is used to calculate the similarity between voiceprint features fea1 and fea2 to obtain a similarity value sim1_2.

[0058] Step 150: In response to the sim1_2 value being greater than the preset sim_threshold value, determine the speaker information based on the voiceprint feature fea1 and the preset voiceprint database; and input the audio segment voc1 and the audio segment voc2 into the speech content recognition model to output the speech content.

[0059] In some embodiments, the sim1_2 value is compared with a preset sim_threshold value. It is necessary to compare the result calculated in step 140 with the preset similarity threshold sim_threshold to determine whether audio segments voc1 and voc2 belong to the same speaker.

[0060] In some embodiments, when sim1_2 is greater than sim_threshold, it is determined that they are from the same person. For example, if the similarity threshold sim_threshold is 0.5 and the value of sim1_2 is 0.8, then audio segments voc1 and voc2 are considered to be from the same speaker. Thus, audio segments determined to be from the same speaker will not be segmented due to silence, and the resulting fused segmentation will make audio segmentation more accurate.

[0061] In some embodiments, the preset voiceprint database includes multiple speaker information and their corresponding voiceprint features; determining the speaker information based on the voiceprint feature fea1 and the preset voiceprint database includes: calculating the sim1_2 value of the voiceprint feature fea1 and each voiceprint feature in the voiceprint database based on the cosine similarity compute_cosine_socre; if the largest sim1_2 value is greater than the sim_threshold value, then the speaker information corresponding to the voiceprint feature with the largest sim1_2 value is taken as the speaker information corresponding to the voiceprint feature fea1.

[0062] In some embodiments, the similarity between the voiceprint feature fea1 and a preset voiceprint database is calculated. Specifically, the sim1_2 values ​​between the voiceprint feature fea1 and multiple voiceprint features in the preset voiceprint database are calculated based on the cosine similarity compute_cosine_socre. If the largest value among the sim1_2 values, i.e. the maximum similarity value max_sim, is less than or equal to sim_threshold, a new speaker information is generated, and the voiceprint database needs to be supplemented. If max_sim is greater than sim_threshold, the speaker information corresponding to this value greater than sim_threshold is taken as the speaker information corresponding to the first voiceprint feature.

[0063] In some embodiments, since audio segments voc1 and voc2 are determined to be from the same speaker, the corresponding speaker information can also be determined by performing the same operation on the voiceprint feature fea2 of audio segment voc2 and a preset voiceprint database. Audio segments voc1 and voc2 are input into a speech content recognition model, and the output is the spoken content. The speech content recognition model adopts a zipformer structure, where the encoder dimension encoder_dims is (384, 384), the attention dimension attention_dim is (256, 256), the encoder mask dimension encoder_unmasked_dims is (256, 256), the downsampling_factors is (2, 4), the encoder layers num_encoder_layers are (12, 12), the feedforward layers are (1536, 2048), and the convolutional module kernel size cnn_module_kernels is (31, 31).

[0064] In some embodiments, the speaker information recorded in the voiceprint database includes the speaker ID and its corresponding voiceprint features, as well as the addition time. Supplementing the voiceprint database requires generating new IDs and recording their corresponding voiceprint features. Of course, the ID is automatically generated by the system, but can also be manually assigned a corresponding name, position, etc., based on the actual situation when generating subsequent speaker information and content. Alternatively, in practice, operators can pre-assign values ​​(e.g., names) to the existing speaker IDs in historical information based on the specific application scenario, and then directly generate the speaker's name, etc.

[0065] Based on the above embodiments, the audio recognition method in another embodiment provided in this disclosure further includes: in response to the sim1_2 value being less than or equal to the sim_threshold value, determining the corresponding speaker information based on the voiceprint feature fea1 and a preset voiceprint database; inputting the audio segment voc1 into a speech content recognition model and outputting the corresponding speech content; continuing to perform audio recognition processing on the audio segment voc2 until all audio segments are recognized, and outputting the corresponding speaker information and speech content.

[0066] In some embodiments, if the sim1_2 value is less than or equal to the sim_threshold value, for example, if the sim1_2 value = 0.4 and the sim_threshold value = 0.5, then audio segment voc1 and audio segment voc2 are not from the same speaker. In this case, speaker information matching and speech content recognition are performed on audio segment voc1 first, and then audio segment voc2 and all subsequent audio segments are further processed for audio recognition until all audio segments to be recognized are recognized. Finally, the speaker information and speech content of each audio segment are obtained.

[0067] In some embodiments, the recognition can be performed while the output is being performed.

[0068] Based on the above embodiments, in another embodiment of the audio recognition method provided in this disclosure, the audio segment includes timestamp information; the method further includes: generating corresponding start and end times for the output speaker information and speech content based on the timestamp information.

[0069] In some embodiments, the start and end times of the corresponding audio segment, i.e., the speaking time period, can be generated based on the timestamp information of the audio segment. For silent segments, no start and end times are generated. The output of the speaking time period can be performed concurrently with the recognition of speaker information and speaking content, or it can be output uniformly after the final recognition is completed.

[0070] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0071] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0072] Figure 2 A block diagram of an audio recognition device 200 according to an embodiment of the present disclosure is shown. Figure 2 As shown, the device 200 includes:

[0073] The audio acquisition and recognition module 210 is used to acquire the audio to be recognized; input the audio to be recognized into the silence detection model, and output multiple audio segments;

[0074] The silence determination module 220 is used to determine the silence segment based on the audio segment and a window detection algorithm based on consecutive frames;

[0075] The voiceprint extraction module 230 is used to extract audio segments voc1 and voc2 based on the silent segment, input the voiceprint feature extraction model, and output the voiceprint feature fea1 of the audio segment voc1 and the voiceprint feature fea2 of the audio segment voc2.

[0076] The voiceprint calculation module 240 is used to calculate the sim1_2 values ​​of the voiceprint feature fea1 and the voiceprint feature fea2 based on the cosine similarity compute_cosine_socre.

[0077] The content recognition module 250 is used to determine speaker information based on the voiceprint feature fea1 and a preset voiceprint database in response to the sim1_2 value being greater than a preset sim_threshold value; and to input the audio segment voc1 and the audio segment voc2 into the speech content recognition model and output the speech content.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0080] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0081] Electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in ROM 302 or a computer program loaded into RAM 303 from storage unit 308. RAM 303 can also store various programs and data required for the operation of electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via bus 304. I / O interface 305 is also connected to bus 304.

[0082] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0083] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as the audio recognition method. For example, in some embodiments, the audio recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the audio recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the audio recognition method by any other suitable means (e.g., by means of firmware).

[0084] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0085] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0088] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0089] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0090] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An audio recognition method, characterized in that, include: Obtain the audio to be recognized; The audio to be identified is input into the silence detection model, which outputs multiple audio segments. Based on the audio segment, a window detection algorithm based on consecutive frames is used to determine the silent segment; Extract the audio segment before the silent segment as audio segment voc1, and extract the audio segment after the silent segment as audio segment voc2. Input the voiceprint feature extraction model and output the voiceprint feature fea1 of audio segment voc1 and the voiceprint feature fea2 of audio segment voc2. The sim1_2 values ​​of the voiceprint features fea1 and fea2 are calculated based on the cosine similarity compute_cosine_socre. In response to the sim1_2 value being greater than the preset sim_threshold value, the speaker information is determined based on the voiceprint feature fea1 and the preset voiceprint database; and the audio segment voc1 and the audio segment voc2 are input into the speech content recognition model to output the speech content.

2. The method according to claim 1, characterized in that, The process involves inputting the audio to be identified into a silence detection model, which then outputs multiple audio segments, including: The VAD model is used to detect silent endpoints and identify silent states in the audio to be identified. Based on the endpoint detection results, the audio to be identified is marked with boundaries to obtain multiple audio segments.

3. The method according to claim 2, characterized in that, The audio segment is an audio fragment consisting of consecutive frames; The step of determining the silent segment based on the audio segment and a window detection algorithm for consecutive frames includes: The continuous frame-based window detection algorithm counts the number of frames in the audio segments marked as silent, and determines the silent segments based on the number of continuous frames.

4. The method according to claim 1, characterized in that, The preset voiceprint database includes multiple speaker information and their corresponding voiceprint features; The step of determining speaker information based on the voiceprint feature fea1 and a preset voiceprint database includes: The voiceprint feature fea1 and the sim1_2 value of each voiceprint feature in the voiceprint database are calculated based on the cosine similarity compute_cosine_socre. If the largest sim1_2 value is greater than the sim_threshold value, then the speaker information corresponding to the voiceprint feature corresponding to the largest sim1_2 value is used as the speaker information corresponding to the voiceprint feature fea1.

5. The method according to claim 1, characterized in that, The method further includes: In response to the sim1_2 value being less than or equal to the sim_threshold value, the corresponding speaker information is determined based on the voiceprint feature fea1 and a preset voiceprint database; and the audio segment voc1 is input into the speech content recognition model to output the corresponding speech content. The audio segment voc2 is further processed for audio recognition until all audio segments are recognized, and the corresponding speaker information and speech content are output.

6. The method according to claim 5, characterized in that, The audio segment includes timestamp information; The method further includes: Based on the timestamp information, corresponding start and end times are generated for the output speaker information and speech content.

7. An audio recognition device, characterized in that, include: The audio acquisition and recognition module is used to acquire the audio to be recognized; The audio to be identified is input into the silence detection model, which outputs multiple audio segments. A silence determination module is used to determine a silence segment based on the audio segment and a window detection algorithm based on consecutive frames. The voiceprint extraction module is used to extract the audio segment before the silence segment as audio segment voc1 and the audio segment after the silence segment as audio segment voc2. It inputs the voiceprint feature extraction model and outputs the voiceprint feature fea1 of the audio segment voc1 and the voiceprint feature fea2 of the audio segment voc2. The voiceprint calculation module is used to calculate the sim1_2 values ​​of the voiceprint features fea1 and fea2 based on the cosine similarity compute_cosine_socre. The content recognition module is used to determine the speaker information based on the voiceprint feature fea1 and the preset voiceprint database in response to the sim1_2 value being greater than the preset sim_threshold value; and to input the audio segment voc1 and the audio segment voc2 into the speech content recognition model and output the speech content.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images