Data annotation method, device, electronic device and readable medium

Abstract

The present application provides a data annotation method, device, electronic device and readable medium. The method includes: obtaining audio data to be detected, the audio data to be detected includes sentences containing target words; performing target word recognition on the audio data to be detected to obtain the timestamp of the target word in the audio data to be detected; performing audio segmentation on the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected; generating annotation data according to the audio segments corresponding to the timestamp of the target word in the audio segments of each sentence. The method can accurately extract the corpus data of the sentence where the target word is located, improve the relevance of the annotation data with the target word, thereby improving the effectiveness of the annotation data, and further facilitating improving the training efficiency of the model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data annotation method, device, electronic device, and readable medium. Background Art

[0002] With the development of artificial intelligence technology, machine learning models are often used to identify and process the content of audio or multimedia resources including audio. The training process of machine learning models usually requires a large amount of annotated data as training data.

[0003] In the related art, the method of generating annotated data is generally to annotate the audio data according to target words included in a section of audio data.

[0004] However, the labeled data obtained through the above labeling process usually includes a large number of sentences that do not contain the target words, which reduces the effectiveness of the labeled data and thus affects the training efficiency of the model. Summary of the Invention

[0005] Based on the above technical problems, the present application provides a data annotation method, device, electronic device and readable medium to accurately extract the corpus data of the sentence where the target word is located, improve the relevance of the annotated data and the target word, thereby improving the effectiveness of the annotated data, and further helping to improve the training efficiency of the model.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a data labeling method is provided, comprising:

[0008] Acquiring audio data to be detected, wherein the audio data to be detected includes a sentence containing a target word;

[0009] Performing target word recognition on the audio data to be detected to obtain a timestamp of the target word in the audio data to be detected;

[0010] Performing audio segmentation on the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected;

[0011] Generate annotation data based on the audio segments corresponding to the timestamps of the target words in the audio segments of the respective sentences.

[0012] According to another aspect of an embodiment of the present application, a data tagging device is provided, comprising:

[0013] A data acquisition module is used to acquire audio data to be detected, wherein the audio data to be detected includes a sentence containing a target word;

[0014] a target word recognition module, configured to perform target word recognition on the audio data to be detected, and obtain a timestamp of the target word in the audio data to be detected;

[0015] An audio segmentation module, configured to segment the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected;

[0016] The annotation data generation module is used to generate annotation data according to the audio segments corresponding to the timestamps of the target words in the audio segments of the respective sentences.

[0017] In some embodiments of the present application, based on the above technical solution, the data acquisition module includes:

[0018] A candidate audio acquisition unit, configured to acquire at least one candidate audio data;

[0019] An audio classification unit, configured to perform audio classification on the at least one candidate audio data using an audio classification model to obtain a scene label and a scene score corresponding to each candidate audio data;

[0020] The audio screening unit is configured to screen out, from the at least one candidate audio data, audio data whose scene label is a speech scene and whose score is higher than a preset target score as the audio data to be detected, based on the scene label and the scene score.

[0021] In some embodiments of the present application, based on the above technical solution, the target word recognition module includes:

[0022] A feature extraction unit, configured to extract features from the audio data to be detected to obtain speech features;

[0023] a target word detection unit, configured to detect the target word in the audio data to be detected based on a preset acoustic model and the speech feature, and determine an audio frame containing the target word in the audio data to be detected;

[0024] A timestamp determination unit is used to determine the timestamp of the target word according to the playback time of the audio frame in the audio data to be detected.

[0025] In some embodiments of the present application, based on the above technical solution, the audio segmentation module includes:

[0026] a text conversion unit, configured to convert the audio data to be detected into text to obtain the text content of the sentence in the audio data to be detected;

[0027] A segmentation unit is used to segment the audio data to be detected according to the silence areas in the audio data to be detected using a sentence segmentation model to obtain audio segments corresponding to each sentence, wherein each audio segment corresponds to the text content of a sentence, and the audio segment includes a timestamp of the playback time of the corresponding sentence in the audio data to be detected.

[0028] In some embodiments of the present application, based on the above technical solution, the annotation data generation module includes:

[0029] a segment determination unit, configured to determine an audio segment corresponding to a sentence containing the target word based on a mapping relationship between the timestamp of the target word and the timestamps of the audio segments of each sentence;

[0030] The annotation data generating unit is configured to generate the annotation data according to the audio segment corresponding to the sentence, the text content, the target word and the timestamp of the audio segment if the text content of the sentence contains the target word.

[0031] In some embodiments of the present application, based on the above technical solution, the data labeling device further includes:

[0032] a sentence-to-be-trained determining unit, configured to determine, based on the actual text content of the sentence in the audio data to be detected and the segmented audio segment, a sentence-to-be-trained whose matching degree between the audio segment and the text content is lower than a matching threshold;

[0033] A targeted training unit is used to perform targeted training on the sentence model based on the sentence to be trained and the audio data to be detected.

[0034] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data labeling method in the above technical solution by executing the executable instructions.

[0035] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data labeling method in the above technical solution is implemented.

[0036] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data labeling method provided in the various optional implementations described above.

[0037] In an embodiment of the present application, the timestamp of the target word is identified from the audio data to be detected, and the audio data to be detected is segmented to obtain audio segments of each sentence in the audio data. Finally, the annotation data is generated based on the audio segments corresponding to the timestamps of the target word. Through the above method, the audio information corresponding to the sentences unrelated to the target word can be removed from the audio data to be detected, and the corpus data of the sentence containing the target word can be accurately extracted. The relevance of the annotation data to the target word is improved, thereby improving the effectiveness of the annotation data, which in turn helps to improve the training efficiency of the model.

[0038] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0040] In the attached figure:

[0041] Figure 1 This is an example diagram of an implementation environment involved in this application;

[0042] Figure 2 This is a schematic flow chart of the overall solution process in the embodiment of the present application;

[0043] Figure 3 is a schematic flow chart of a data annotation method in an embodiment of the present application;

[0044] Figure 4 This is a flowchart of audio classification in an embodiment of the present application;

[0045] Figure 5 A schematic diagram showing the audio processing flow in an embodiment of the present application is shown;

[0046] Figure 6 This is another example flow chart of the data annotation method in the embodiment of the present application;

[0047] Figure 7 The following schematically shows a block diagram of the data annotation device in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0050] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0051] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0052] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0053] It should be understood that the present application can be applied to the field of artificial intelligence, and specifically to the annotation scenario of speech audio. In the scheme of the present application, before the audio data is annotated, the audio is preprocessed, including target word recognition of the audio data according to the identified target word, obtaining the timestamp of the target word in the audio data, and sentence segmentation of the audio data to obtain the timestamp or sentence information of each sentence, and then selecting the corresponding sentence according to the timestamp of the target word, and cutting out the audio segment corresponding to the sentence from the audio data for annotation, so as to obtain data with a stronger correlation with the target word. In the scheme of the present application, target word recognition is performed by waking up the target word, and audio sentence segmentation is performed by automatic speech recognition technology combined with voice activity detection.

[0054] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0055] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0056] Key technologies in speech technology include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech becoming one of the most promising methods of human-computer interaction.

[0057] See also Figure 1 , Figure 1 This is an example diagram of an implementation environment involved in this application. Figure 1 As shown, the implementation environment includes a terminal 110 and a server 120. Terminal 110 and server 120 have pre-established a communication connection for data exchange. Terminal 110 runs an audio annotation program that provides an audio annotation interface for displaying the information required for audio annotation. Server 120 is used to provide data services for the audio annotation program running on terminal 110. For example, terminal 110 needs to obtain the information required for audio annotation from server 120 for display, and the audio annotation corpus generated by terminal 110 is stored on server 120.

[0058] It should be noted that the terminal 110 can be a terminal device such as a computer, a vehicle-mounted terminal, a laptop computer, a tablet computer, etc., and the server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, big data and artificial intelligence platforms.

[0059] Figure 2It is a schematic flow chart of the overall solution process in the embodiment of the present application. It can be understood that the process can be executed by the above-mentioned terminal or by the server. Specifically, the input audio is first received in step 201, and then the audio is input into the audio classification service in step 202 for scene recognition and classification, thereby obtaining audio with scene labels and scores, wherein the scene labels are used to mark the scenes of the content of the audio data, such as language, songs, crowds, interference sounds, etc. For the obtained audio with scene labels and scores, in step 203, the audio with a score greater than a predetermined score (for example, 80 points) and a scene label of speech is filtered out, and the audio that does not meet the conditions can be discarded. For the filtered audio, it can be input into the target word wake-up service in step 204 and into the audio ASR segmentation service in step 206. In step 204, target word recognition can be performed according to the customized target word, so that in step 205, the audio, target word and the timestamp a corresponding to the target word are obtained as the output result. In step 205, the audio can be segmented based on silent regions within the audio, resulting in step 207 as an output result containing the audio, audio segments, text content converted from the audio, and timestamp b. In step 208, based on the results of steps 205 and 207, it is determined whether timestamp b contains timestamp a. If not, it is discarded. If so, in step 209, the text content containing the video, the target word, the sentence containing the target word, and the corresponding timestamp b is obtained as an output result, allowing for data annotation.

[0060] The following is a detailed description of the technical solution provided by this application in conjunction with specific implementation methods. Figure 3 , Figure 3 This is a schematic flow chart of the data labeling method in the embodiment of the present application. The data labeling method of the present application will be described below using the terminal as the execution subject. It is understandable that the method can also be executed by the server. The data labeling method includes the following steps S310 to S340:

[0061] Step S310: Acquire audio data to be detected, where the audio data to be detected includes a sentence containing a target word.

[0062] Specifically, the audio data to be detected can be data crawled from the target data to be detected using a crawler based on the intended use of the data. For example, the URL to be crawled can be entered into a crawling framework based on the training scope required for the service. A crawler using the Python programming language, using the Scrapy framework, can be used to request the URL using the Goose3 library, retrieve the corresponding HTML information, and then crawl the audio information. After the crawled information is determined to be the target word to be detected, the crawled audio information can be initially screened to obtain audio data containing the target word as the audio data to be detected.

[0063] In one embodiment of the present application, based on the above-mentioned technical solution, the process of obtaining the audio data to be detected may include the following steps: obtaining at least one candidate audio data; performing audio classification on the at least one candidate audio data through an audio classification model to obtain the scene label and scene score corresponding to each candidate audio data; based on the scene label and the scene score, filtering out the audio data whose scene label is a speech scene and whose score is higher than a preset target score from the at least one candidate audio data as the audio data to be detected.

[0064] Specifically, the process of obtaining at least one candidate audio data is to obtain it from the data source by crawling. The audio classification model is a pre-trained model used to label and classify audio scenes. Audio scene labels can include various types of content such as speech, noise, music, songs, nature, etc. The audio classification model also outputs a score for the audio data, which can indicate the degree of match between the audio data and the output scene. A score higher than the preset target score indicates that the audio data has sufficient relevance to the corresponding scene, and can be used to generate labeled data. Please refer to Figure 4 , Figure 4 This is a flow chart illustrating the audio classification process in an embodiment of the present application. As shown, data from a data source is classified to produce audio with scene labels such as speech, noise, music, and song. When the target scene is speech, the audio data for the speech scene is retained, while audio data for other scenes is discarded.

[0065] Step S320 , performing target word recognition on the audio data to be detected, and obtaining a timestamp of the target word in the audio data to be detected.

[0066] Target word recognition can be performed through a machine model or through a recognition algorithm, or it can be performed through voiceprint analysis and intonation analysis based on the audio corresponding to the target word. During the analysis process, adjacent words associated with the target word can also be included for joint analysis. The timestamp of the target word usually corresponds to the time when the target word is played in the audio data to be detected. For example, the target word is "holiday" and the audio data to be detected is "I have forgotten that I will have a holiday starting tomorrow". Assuming that the audio of the word "holiday" in the audio data to be detected is played from the 34th frame to the 40th frame, the timestamp of the target word can be the 34th frame. It can be understood that the timestamp can be expressed in various units, such as seconds, milliseconds, frames, etc., and this application does not limit the representation method and accuracy.

[0067] In one embodiment of the present application, based on the above-mentioned technical solution, the process of identifying the target word in the audio data to be detected and obtaining the timestamp of the target word in the audio data to be detected may include the following steps: extracting features from the audio data to be detected to obtain speech features; detecting the target word in the audio data to be detected based on a preset acoustic model and the speech features, and determining the audio frame of the target word in the audio data to be detected; and determining the timestamp of the target word based on the playback time of the audio frame in the audio data to be detected. The preset acoustic model is a pre-trained machine learning model that identifies whether the audio data contains the target word and the time position of the target word in the audio data based on the speech features of the audio data to be detected. In one embodiment, the preset acoustic model also uses the speech features of the target word as data, so that it can match and identify various target words, or the preset acoustic model converts the sentences in the audio data into text and then matches and searches with the target words. In a specific example, the audio data to be detected is a voice recording, which includes audio data of multiple sentences. When obtaining the timestamp of the target word, the voice recording will first be subjected to feature extraction, such as converting the voice audio into a frequency domain representation, such as converting and calculating the voice audio using Mel cepstral coefficients, linear prediction coefficients, linear prediction cepstral coefficients or line spectrum frequencies, thereby obtaining the voice features of the voice audio. The voice features and the target word to be detected will be input into a trained preset acoustic model for calculation. Specifically, depending on the training method of the preset acoustic model, the target word can be input in the form of text or in the form of audio features. Taking text input as an example, when training the preset acoustic model, the data used for training is the extracted voice features, the text information of the target word, whether the audio includes the annotation information of the target word, and the timestamp of the target word in the audio. The preset acoustic model will learn the corresponding relationship in the training data, thereby establishing a corresponding relationship between the voice features and the text information and timestamp of the target word. In application, assuming the speech recording is a 3-second speech with the content "Heavy rain and strong winds may occur in XX area this morning," and the target word is "heavy rain" appearing at the 2500th millisecond, the speech features of the speech recording and the text content of "heavy rain" are input into the preset acoustic model. The preset acoustic model will then output the result of the speech including the target word "heavy rain" and the timestamp of the word "heavy rain" in the speech audio being 2500 milliseconds. In another embodiment, the preset acoustic model uses the audio information of the target word as input. Still using the above speech content as an example, in application, the speech features and the audio information of the word "heavy rain" are input. This audio information can be audio synthesized by software, and the preset acoustic model will still output the above result.The preset acoustic model can also be implemented in an audio matching manner, for example, by extracting features from the audio of the target word and then matching the audio features of the target word with the speech features to determine whether the target word exists in the speech audio and the specific timestamp.

[0068] Step S330: segment the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected.

[0069] Audio segmentation can be performed by combining speech recognition technology with voice activity detection technology. Speech recognition technology converts the lexical content in the speech into computer-readable input, such as keystrokes, binary codes, or character sequences, and voice activity detection technology is used to identify the boundaries of the speech. Boundaries are identified by identifying pauses, silent periods, silencers, or other audio segments that do not contain speech content in the speech, and the audio data is divided into multiple audio segments based on the identified boundaries. Each audio segment includes a sentence or a phrase. The audio segmentation process can convert the sentences in the audio into text and divide them into individual sentences, and the audio is divided into audio segments corresponding to the sentences based on the silent intervals or pauses in the audio.

[0070] In one embodiment of the present application, based on the above-mentioned technical solution, the process of performing audio segmentation on the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected may include the following steps: performing text conversion on the audio data to be detected to obtain the text content of the sentence in the audio data to be detected; segmenting the audio data to be detected according to the silent area in the audio data to be detected through a sentence segmentation model to obtain audio segments corresponding to each sentence, wherein each audio segment corresponds to the text content of a sentence, and the audio segment includes a timestamp of the playback time of the corresponding sentence in the audio data to be detected.

[0071] Specifically, text conversion can be performed using an ASR (Analog Speech Recognition) algorithm. For example, the audio data to be tested includes the speech content of "Are you all free this week? Let's get together and find a place to eat." Using the ASR algorithm, the speech content can be converted into corresponding text content. Subsequently, silences in the audio data to be tested are identified. Typically, during verbal communication, there are silences between sentences where no speech content exists. These silences can be used to separate sentences. Using the silences, the three sentences in the audio data to be tested are divided into three parts: "Are you all free this week?", "Let's get together?", and "Let's find a place to eat." Each audio segment corresponds to the text content of a sentence. During the segmentation process, the start time of each audio segment within the entire audio data to be segmented is recorded as the audio segment's timestamp. For example, assuming the total length of the audio is 4 seconds, the first second contains no speech content, and each sentence lasts 1 second, the timestamps of the three audio segments might be the 2nd, 3rd, and 4th seconds, respectively. It is understandable that the basis for dividing sentences can also be other contents, such as learning commonly used spoken language through processed audio data as the basis for dividing audio segments. The recognition standard of the silent area can also be determined according to the specific application scenario or according to the speech content. For example, the speaking speed in the audio to be detected can be identified, and then the duration standard of the silent area can be further determined based on the speaking speed. For speech audio with a slower speaking speed, the duration corresponding to the silent area can be extended, while for content such as advertising broadcasts with a faster speaking speed, the silent area can be relatively shortened.

[0072] Step S340 : Generate annotation data based on the audio segments of the respective sentences that have timestamps corresponding to the target words.

[0073] The audio segments divided according to the audio data include the speech content in the audio data. Therefore, the identified target word will be in at least one audio segment. The audio segments are divided from the audio data, so each audio segment will correspond to a part of the playback time in the audio data. For example, a sentence may last from the 30th frame to the 50th frame. If the playback time covered by an audio data includes the timestamp corresponding to the target word, then the audio segment can be considered to be an audio segment corresponding to the timestamp of the target word. The method of generating labeled data can usually be performed by a trained machine model. Specifically, each character of the text information of the audio segment is labeled to mark the position of the target word and other content for training.

[0074] In one embodiment of the present application, based on the above-mentioned technical solution, the generation of annotation data based on the audio segments corresponding to the timestamps of the target words in the audio segments of the various sentences includes: determining the audio segment corresponding to the sentence containing the target word based on the mapping relationship between the timestamp of the target word and the timestamps of the audio segments of the various sentences; and if the text content of the sentence contains the target word, generating the annotation data based on the audio segment corresponding to the sentence, the text content, the target word, and the timestamp of the audio segment. Specifically, the timestamp of the target word marks the time when the target word is played in the entire audio data to be detected, while the timestamp of the audio segment indicates the time when each sentence in the audio data to be detected begins to play. It will be understood that when the audio data to be detected contains the target word, the target word will appear in one or more sentences, and therefore, the target word will also correspond to one or more audio segments. Based on the timestamps of both, the audio segment containing the target word can be determined. Taking the three audio clips of audio clip A "Is everyone free this week?", audio clip B "Let's get together", and audio clip C "Let's find a place to eat" as examples, assuming that the timestamps of the three audio clips are the 1st second, 2nd second, and 3rd second respectively, and the target word is "everyone", the determined timestamps of the target word are 1.3 seconds and 2.1 seconds. Based on the correspondence between 1.3 seconds and 1 second and the correspondence between 2.1 and 2 seconds, the audio clips corresponding to the target word "everyone" can be determined to be audio clip A and audio clip B. Subsequently, based on the previously transferred text content corresponding to audio clip A and audio clip B, it can be determined that the text content does contain the target word "everyone", so that annotation data can be generated based on audio clips A and B, the corresponding text content, the target word "everyone" and the timestamps of the audio clips, that is, the annotation data contains information about the target word "everyone", audio clip A, the text content of audio clip A "Is everyone free this week" and the timestamp 1 second and audio clip B, the text content of audio clip B "Let's get together", and the timestamp 2 second of audio clip B.

[0075] See also Figure 5 , Figure 5 FIG. 1 shows a schematic diagram of the audio processing flow in an embodiment of the present application. Figure 5 As shown, the input audio is input into the target word wake-up service and the ASR segmentation service after audio classification, and the timestamps of the target word and the timestamps of the audio segments of each sentence are obtained respectively. Then, timestamp matching is performed to obtain the audio segment including the target word.

[0076] In one embodiment of the present application, based on the above-mentioned technical solution, after segmenting the audio data to be detected based on the silence regions in the audio data to be detected using a sentence segmentation model to obtain audio segments corresponding to the textual content of each sentence, the method further includes the following steps: determining, based on the actual textual content of the sentence in the audio data to be detected and the segmented audio segments, sentences to be trained whose matching degree between the audio segments and the textual content is below a matching threshold; and performing targeted training on the sentence segmentation model based on the sentences to be trained and the audio data to be detected. Specifically, during use, the matching degree between the audio segments and the textual content can be determined based on the actual textual content of the audio data to be detected and the segmented audio segments. If the matching degree is below a predetermined threshold, it indicates that the sentence segmentation model is not effective for the sentence, and therefore, the sentence is determined as a sentence to be trained. Based on the sentences to be trained, the audio data to be detected, and the correct segmentation results, the sentence segmentation model can be trained to improve the sentence segmentation performance of the sentence model. Specifically, audio data with a matching degree below a predetermined threshold and the corresponding actual textual content are collected. Based on the content of the audio data, the audio data can be manually segmented to form audio segments. The manually segmented audio segments and actual text content are used as the target results of the training data, along with the original audio data, to form the training data. Subsequently, the training data is processed by a sentence segmentation model to obtain output audio segments and text content. A loss function is used to calculate the loss between the audio segments and text content output by the sentence segmentation model and the target results, and the sentence model parameters are adjusted based on the loss value. Training is repeated until the loss between the results output by the sentence segmentation model and the target results meets the termination criteria, or the number of training times meets the termination criteria. For example, for the audio data "XXX is really sober in the world, so admirable", the recognition of the internet slang "sober in the world" may result in a recognition error due to the segmentation error caused by segmenting the sentence into "XXX is really sober in the world", "sober", and "so admirable". In this case, the audio data can be manually segmented and used together with the corresponding real text content to train the sentence segmentation model, so that the sentence segmentation model can correctly segment the audio data and subsequent audio data containing similar words and phrases. Since the data used for training is audio data that the sentence segmentation model has found to have poor or inaccurate sentence segmentation effects in actual applications, collecting these audio data as training data to conduct targeted training on the sentence model can make the sentence segmentation results of the sentence model for these audio data and similar audio data more accurate, thereby obtaining a sentence model with more accurate segmentation results for the audio data used for training and similar audio data.

[0077] In one embodiment, the audio segmentation process can also be separated from the sentence segmentation process. For example, the retained speech audio data is first transcribed using ASR technology and the transcribed text information is distinguished into sentences. At the same time, the target word wake-up technology is used to recognize the retained speech audio data to obtain the recognized target word result and timestamp, and the threshold is taken as the segmented data where the model recognizes the target word threshold. Finally, the timestamp of the target word result after screening is mapped to the timestamp of the ASR sentence result containing the target word to obtain the timestamp of the short sentence containing the target word. Then, the audio cutting tool is used to directionally cut out the audio information in the timestamp of the final short sentence containing the target word according to the timestamp of the short sentence, thereby obtaining the audio segment corresponding to the timestamp of the target word. Specifically, please refer to Figure 6 , Figure 6 This is another example flow chart of the data annotation method in the embodiment of this application. Figure 6 As shown, speech audio undergoes preprocessing to identify the target word. The segmented data and timestamps are then selected and segmented using the ASR sentence segmentation service. The sentence is then matched with the timestamp of the target word to obtain the sentence containing the target word. Directed segmentation is then performed to obtain the audio segment containing the target word, and data annotation is then performed. Figure 6 The example sentence segmentation process is also shown in Figure 2. The target word is "meeting at work," and the corresponding time segment is from 12.75 seconds to 13.14 seconds. In the ASR sentence segmentation results, the time segment of the second audio segment is from 11 seconds to 20 seconds, which contains the target word. Therefore, this audio segment is split and retained, and it is the audio segment containing the target word.

[0078] In an embodiment of the present application, the timestamp of the target word is identified from the audio data to be detected, and the audio data to be detected is segmented to obtain audio segments of each sentence in the audio data. Finally, the annotation data is generated based on the audio segments corresponding to the timestamps of the target word. Through the above method, the audio information corresponding to the sentences unrelated to the target word can be removed from the audio data to be detected, and the corpus data of the sentence containing the target word can be accurately extracted. The relevance of the annotation data to the target word is improved, thereby improving the effectiveness of the annotation data, which in turn helps to improve the training efficiency of the model.

[0079] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0080] The following introduces the device implementation of the present application, which can be used to execute the data labeling method in the above embodiments of the present application. Figure 7 The following schematically shows a block diagram of the data annotation device in an embodiment of the present application. Figure 7 As shown, the data tagging device 700 may mainly include:

[0081] The data acquisition module 710 is used to acquire audio data to be detected, wherein the audio data to be detected includes a sentence containing a target word;

[0082] a target word recognition module 720, configured to perform target word recognition on the audio data to be detected, and obtain a timestamp of the target word in the audio data to be detected;

[0083] An audio segmentation module 730 is configured to segment the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected;

[0084] The annotation data generating module 740 is configured to generate annotation data based on the audio segments corresponding to the timestamps of the target words in the audio segments of the respective sentences.

[0085] In some embodiments of the present application, based on the above technical solution, the data acquisition module 710 includes:

[0086] A candidate audio acquisition unit, configured to acquire at least one candidate audio data;

[0087] An audio classification unit, configured to perform audio classification on the at least one candidate audio data using an audio classification model to obtain a scene label and a scene score corresponding to each candidate audio data;

[0088] The audio screening unit is used to screen out, from the at least one candidate audio data, audio data whose scene label is a speech scene and whose score is higher than a preset target score as the audio data to be detected based on the scene label and the scene score.

[0089] In some embodiments of the present application, based on the above technical solution, the target word recognition module 720 includes:

[0090] A feature extraction unit, configured to extract features from the audio data to be detected to obtain speech features;

[0091] a target word detection unit, configured to detect the target word in the audio data to be detected based on a preset acoustic model and the speech feature, and determine an audio frame containing the target word in the audio data to be detected;

[0092] A timestamp determination unit is used to determine the timestamp of the target word according to the playback time of the audio frame in the audio data to be detected.

[0093] In some embodiments of the present application, based on the above technical solution, the audio segmentation module 730 includes:

[0094] a text conversion unit, configured to convert the audio data to be detected into text to obtain the text content of the sentence in the audio data to be detected;

[0095] A segmentation unit is used to segment the audio data to be detected according to the silence areas in the audio data to be detected using a sentence segmentation model to obtain audio segments corresponding to each sentence, wherein each audio segment corresponds to the text content of a sentence, and the audio segment includes a timestamp of the playback time of the corresponding sentence in the audio data to be detected.

[0096] In some embodiments of the present application, based on the above technical solution, the annotation data generation module 740 includes:

[0097] a segment determination unit, configured to determine an audio segment corresponding to a sentence containing the target word based on a mapping relationship between the timestamp of the target word and the timestamps of the audio segments of each sentence;

[0098] The annotation data generating unit is configured to generate the annotation data according to the audio segment corresponding to the sentence, the text content, the target word and the timestamp of the audio segment if the text content of the sentence contains the target word.

[0099] In some embodiments of the present application, based on the above technical solution, the data labeling device 700 further includes:

[0100] a sentence-to-be-trained determining unit, configured to determine, based on the actual text content of the sentence in the audio data to be detected and the segmented audio segment, a sentence-to-be-trained whose matching degree between the audio segment and the text content is lower than a matching threshold;

[0101] A targeted training unit is used to perform targeted training on the sentence model based on the sentence to be trained and the audio data to be detected.

[0102] It should be noted that the apparatus provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0103] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0104] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0105] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803. Various programs and data required for system operation are also stored in the RAM 803. The CPU 801, ROM 802 and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0106] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. Removable media 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read therefrom can be installed into the storage section 808 as needed.

[0107] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the various functions defined in the system of the present application are executed.

[0108] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0111] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0112] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0113] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data labeling method, characterized in that: include: Acquire audio data to be detected, wherein the audio data to be detected includes a sentence containing a target word; Performing target word recognition on the audio data to be detected to obtain a timestamp of the target word in the audio data to be detected; Performing audio segmentation on the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected, wherein the audio segments include a timestamp of a playback time of the corresponding sentence in the audio data to be detected; Determining the audio segment corresponding to the sentence containing the target word according to the mapping relationship between the timestamp of the target word and the timestamp of the audio segment of each sentence; If the target word is contained in the text content of the sentence, the annotation data is generated according to the audio segment corresponding to the sentence, the text content, the target word and the timestamp of the audio segment, wherein the text content of the sentence is obtained by converting the audio data to be detected into text.

2. The method according to claim 1, characterized in that The step of obtaining the audio data to be detected includes: Acquire at least one candidate audio data; Performing audio classification on the at least one candidate audio data through an audio classification model to obtain a scene label and a scene score corresponding to each candidate audio data; According to the scene label and the scene score, audio data whose scene label is a speech scene and whose score is higher than a preset target score is screened out from the at least one candidate audio data as the audio data to be detected.

3. The method according to claim 1, characterized in that The step of performing target word recognition on the audio data to be detected to obtain a timestamp of the target word in the audio data to be detected includes: Extracting features from the audio data to be detected to obtain speech features; Detecting the target word in the audio data to be detected according to the preset acoustic model and the speech feature, and determining the audio frame of the target word in the audio data to be detected; The timestamp of the target word is determined according to the playing time of the audio frame in the audio data to be detected.

4. The method according to claim 1, characterized in that The step of segmenting the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected includes: Convert the audio data to be detected into text to obtain the text content of the sentence in the audio data to be detected; The audio data to be detected is segmented according to the silent areas in the audio data to be detected by a sentence segmentation model to obtain audio segments corresponding to each sentence, wherein each audio segment corresponds to the text content of a sentence.

5. The method according to claim 4, characterized in that After segmenting the audio data to be detected according to the silent areas in the audio data to be detected by the sentence segmentation model to obtain audio segments corresponding to the text content of each sentence, the method further includes: Determine, based on the actual text content of the sentence in the audio data to be detected and the audio segment obtained by segmentation, a sentence to be trained whose matching degree between the audio segment and the text content is lower than a matching threshold; The sentence model is trained specifically according to the sentence to be trained and the audio data to be detected.

6. A data labeling device, characterized in that: include: A data acquisition module, used to acquire audio data to be detected, wherein the audio data to be detected includes a sentence containing a target word; A target word recognition module, used to perform target word recognition on the audio data to be detected, and obtain a timestamp of the target word in the audio data to be detected; An audio segmentation module, configured to segment the audio data to be detected to obtain audio segments corresponding to each sentence in the audio data to be detected, wherein the audio segments include a timestamp of a playback time of the corresponding sentence in the audio data to be detected; The annotation data generating module is used to determine the audio segment corresponding to the sentence containing the target word according to the mapping relationship between the timestamp of the target word and the timestamp of the audio segment of each sentence, and if the text content of the sentence contains the target word, the annotation data is generated according to the audio segment corresponding to the sentence, the text content, the target word and the timestamp of the audio segment, wherein the text content of the sentence is obtained by converting the audio data to be detected into text.

7. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the data labeling method described in any one of claims 1 to 5 by executing the executable instructions.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data labeling method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads and executes the computer instructions from the computer-readable storage medium, so that the computer device executes the data labeling method according to any one of claims 1 to 5.

Images