Turing audio recording method, device, medium, computing device and program product

Abstract

Embodiments of the present disclosure provide a song library audio collection method, device, medium, computing device and program product, relating to the field of audio recognition. The song library audio collection method comprises: obtaining a plurality of first audio segments that fail in audio recognition within a first time length, searching for recognition results matched with the first audio segments from a plurality of second audio segments that fail in audio recognition within a second time length, and performing audio recognition based on the recognition results after aggregation and mining, and entering the recognized audio into a song library. The song library audio collection method of the embodiments of the present disclosure can perform aggregation and mining based on the first audio segments, without identifying all audio segments one by one. Through audio recognition of the audio segments after aggregation and mining, the corresponding audio from the source of the audio segments can be quickly obtained and entered into the song library, thereby reducing the workload in the process of collecting audio and improving the efficiency of song library audio collection.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of audio recognition, and more specifically, the embodiments of this disclosure relate to a method, apparatus, medium, computing device, and program product for recording music library audio. Background Technology

[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] With the development of computer technology, many application software programs can provide the function of identifying the source of audio. In particular, some audio-visual playback software provides users with a song recognition function based on an established music recognition library. Users input an audio fragment containing song content, and the software background matches the corresponding song in the music recognition library based on the audio fragment and returns it to the user.

[0004] In scenarios involving music recognition and identifying the source of audio, the completeness of the music library is a key factor affecting the success rate of recognition. If the audio in the library does not cover the songs the user wants to identify, recognition will fail. Therefore, how to include audio in the music library to meet users' recognition needs is a major problem that urgently needs to be solved. Summary of the Invention

[0005] This disclosure provides a method, apparatus, medium, computing device, and program product for recording audio in a music library, so as to improve the efficiency of recording audio in a music library and increase the success rate of audio recognition.

[0006] In a first aspect of this disclosure, a method for recording music library audio is provided, comprising:

[0007] Retrieve multiple first audio segments that failed to be recognized within the first duration;

[0008] For each first audio segment, among multiple second audio segments that fail to be recognized within a second duration, a recognition result matching the first audio segment is retrieved, wherein the recognition result includes at least one second audio segment, and the first duration is less than the second duration;

[0009] The recognition results corresponding to the multiple first audio segments are aggregated and mined, and audio recognition is performed based on the aggregated mining results. The recognized audio is then recorded into the music library. The aggregated mining results include at least one second audio segment.

[0010] In another embodiment of this disclosure, the step of aggregating and mining the recognition results corresponding to the plurality of first audio segments respectively, and performing audio recognition based on the aggregating and mining results, includes:

[0011] The recognition results containing the same second audio segment are merged to obtain a target audio segment set, which contains all the second audio segments in the recognition results corresponding to at least two first audio segments respectively;

[0012] Based on the second audio segment in the target audio segment set, audio recognition is performed using a preset audio recognition method to obtain the recognized audio;

[0013] The preset audio recognition method includes: performing audio recognition on at least one audio segment in the audio segment set through manual listening; or, splicing together at least two second audio segments in the target audio segment set, and using the spliced ​​audio segment to match the corresponding audio in the music library.

[0014] In another embodiment of this disclosure, merging the recognition results containing the same second audio segment to obtain a target audio segment set includes:

[0015] From the recognition results corresponding to the plurality of first audio segments respectively, determine whether any two recognition results meet the merging condition, and merge the recognition results that meet the merging condition. The merged recognition result includes all the second audio segments in the recognition results before merging.

[0016] The merged recognition result and other recognition results that meet the merging conditions are merged until no recognition result meets the merging conditions, thus obtaining a set of target audio segments;

[0017] The merging condition is that the number of second audio segments in the intersection of the two recognition results is greater than or equal to a preset first threshold.

[0018] In another embodiment of this disclosure, the step of performing audio recognition based on a second audio segment in the target audio segment set to obtain the recognized audio includes:

[0019] The set of target audio segments to be identified is selected from multiple sets of target audio segments, wherein the selection includes a first selection and / or a second selection;

[0020] Audio recognition is performed on audio segments in the set of target audio segments to be identified, and the identified audio is obtained. The audio recognition includes manual recognition.

[0021] The first filtering includes: arranging the multiple target audio segment sets in descending order of the number of audio segments they contain, and retaining the first K target audio segment sets after the first filtering as the target audio segment set, where K is a positive integer;

[0022] The second filtering includes: determining the number of audio segments contained in any target audio segment set among the plurality of target audio segment sets; if the number of audio segments contained in the target audio segment set is greater than or equal to a preset second threshold, then the target audio segment set is retained as the target audio segment set after the second filtering.

[0023] In another embodiment of this disclosure, retrieving a recognition result matching the first audio segment from among a plurality of second audio segments that failed to be recognized within the second duration includes:

[0024] Extract the audio fingerprint of the first audio segment;

[0025] In the second time period, among the multiple second audio segments that failed to be recognized, the number of audio fingerprint matches between the audio fingerprint of the first audio segment and the audio fingerprint of the second audio segment is determined based on the audio fingerprint of the first audio segment and the audio fingerprint of the second audio segment.

[0026] If the number of audio fingerprint hits is greater than or equal to a preset third threshold, the recognition result of the first audio segment is generated based on the second audio segment.

[0027] In another embodiment of this disclosure, extracting the audio fingerprint of the first audio segment includes:

[0028] Perform a short-time Fourier transform on the first audio segment to obtain the spectrogram of the first audio segment;

[0029] An audio fingerprint of the first audio segment is generated based on the local peak points in the spectrogram of the first audio segment.

[0030] In another embodiment of this disclosure, the audio recognition based on the results of aggregation mining includes:

[0031] First auxiliary information is generated based on the second audio segment in the results of the aggregated mining;

[0032] The second audio segment is subjected to audio recognition based on the first auxiliary information to obtain the recognized audio. The first auxiliary information includes the pitch curve of the second audio segment and / or the text information obtained after speech recognition of the second audio segment.

[0033] In another embodiment of this disclosure, the audio recognition based on the results of aggregation mining includes:

[0034] Obtain second auxiliary information, which is the voice and / or text information input by the user after failing to recognize the first audio segment;

[0035] Based on the second auxiliary information, the second audio segment in the result of the aggregated mining is subjected to audio recognition to obtain the recognized audio.

[0036] In a second aspect of this disclosure, a music library audio recording device is provided, comprising:

[0037] The acquisition module is used to acquire multiple first audio segments that failed to be recognized within the first duration.

[0038] The retrieval module is configured to, for each first audio segment, retrieve a recognition result matching the first audio segment from among a plurality of second audio segments that failed to be recognized within a second duration, wherein the recognition result includes at least one second audio segment and the first duration is less than the second duration;

[0039] The mining and recognition module is used to aggregate and mine the recognition results corresponding to the multiple first audio segments respectively, and to perform audio recognition based on the aggregate mining results, and to record the recognized audio into the music library. The aggregate mining results include at least one second audio segment.

[0040] In a third aspect of the present disclosure, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of the first aspects.

[0041] In a fourth aspect of this disclosure, a computing device is provided, comprising: at least one processor;

[0042] and a memory communicatively connected to the at least one processor;

[0043] The memory stores instructions executable by the at least one processor to cause the computing device to perform the method as described in any one of the first aspects.

[0044] In a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in any of the first aspects.

[0045] According to the music library audio recording method, apparatus, medium, computing device, and program product of this disclosure, by acquiring multiple first audio segments that failed audio recognition within a first time period, recognition results matching the first audio segments can be retrieved from multiple second audio segments that failed audio recognition within a second time period. Furthermore, based on the recognition results, aggregation mining can be performed before audio recognition, and the recognized audio can be entered into the music library. The music library audio recording method of this disclosure can perform aggregation mining based on first audio segments, eliminating the need to recognize all audio segments individually. By performing audio recognition on the aggregated and mined audio segments, the audio source of the corresponding audio segment can be quickly obtained and entered into the music library, thereby reducing the workload in the audio recording process and improving the efficiency of music library audio recording. Attached Figure Description

[0046] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0047] Figure 1 The illustration schematically shows an application scenario diagram of the music library audio recording method according to an embodiment of the present disclosure;

[0048] Figure 2 A schematic flowchart of a method for recording music library audio according to an embodiment of the present disclosure is shown.

[0049] Figure 3a A schematic flowchart illustrating an example of audio recognition according to an embodiment of the present disclosure is shown.

[0050] Figure 3b A schematic flowchart of another example of audio recognition according to an embodiment of the present disclosure is shown;

[0051] Figure 4 A schematic flowchart of an audio recognition method according to an embodiment of the present disclosure is shown.

[0052] Figure 5 A schematic flowchart illustrating another example of audio recognition according to an embodiment of the present disclosure is shown.

[0053] Figure 6 A schematic flowchart of a method for obtaining identification results according to an embodiment of the present disclosure is shown.

[0054] Figure 7 A schematic flowchart of another audio recognition method according to an embodiment of the present disclosure is shown;

[0055] Figure 8A schematic flowchart of another audio recognition method according to an embodiment of the present disclosure is shown;

[0056] Figure 9 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown.

[0057] Figure 10 A schematic diagram of the structure of a music library audio recording device according to an embodiment of the present disclosure is shown.

[0058] Figure 11 A schematic diagram of a computing device according to an embodiment of the present disclosure is shown.

[0059] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0060] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0061] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0062] According to embodiments of this disclosure, a method, apparatus, medium, computing device, and program product for recording music library audio are proposed.

[0063] In this article, it is important to understand that the terms used have the following meanings:

[0064] Song recognition is a recognition technology in the field of audio information retrieval. This technology requires establishing a song recognition database containing a number of songs. When a song recognition requirement arises, the corresponding song can be retrieved from the database based on the audio segment of the song to be identified.

[0065] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0066] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0067] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0069] With the development of internet technology, the amount of audio has increased rapidly, leading to a growing demand for audio recognition. To address this, some music software offers users features such as song recognition to identify the source of the audio. Unlike speech recognition, which converts speech into text, song recognition is a sound-to-sound search. A typical song recognition process might involve: for audio whose source is unidentifiable, the user records an audio segment using the software client's recording function. The client sends the audio segment to the server, which then searches its pre-built music library for a matching audio segment and returns it to the client. The client then displays the complete audio segment to the user.

[0070] The inventors have discovered that while audio recognition functions such as song recognition can address some needs for identifying audio sources, their success rate still needs improvement. Especially with the development of short videos and other fields, popular songs often quickly attract a large number of users, generating numerous requests for source identification. However, the success rate of this identification is low, with many instances of failure.

[0071] Further research by the inventors revealed that the main reason for audio source recognition failure lies in an incomplete music library. Successful audio source recognition requires that the music library contains the audio the user wants to identify. Especially when a large number of recognition requests are added in a short period, multiple requests may correspond to the same song. If the song is not included in the software's music library, all these recognition requests will fail, severely impacting the success rate of audio source recognition. To address the issue of incomplete music library content, some software updates the library manually by collecting and adding audio not currently included. However, this method requires a large number of personnel, is labor-intensive, and cannot differentiate between new requests for the same audio source within a short period, resulting in low efficiency.

[0072] The inventors thus conceived of collecting audio segments that users failed to identify, processing them through clustering and other methods, grouping audio segments corresponding to the same audio source into a single group, and then identifying the audio segments within each group to add the identified audio to the music library. This method eliminates the need to identify all audio segments individually; by identifying only a subset of segments within each group, the corresponding audio can be added, thereby reducing the workload in the audio addition process and improving the efficiency of music library audio addition.

[0073] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0074] Application Scenarios Overview

[0075] First refer to Figure 1 , Figure 1 This is a schematic diagram illustrating one application scenario provided in this disclosure. For example... Figure 1 As shown, terminal 102 communicates with server 101 via a network. A data storage system can store the data that server 101 needs to process. The data storage system can be integrated onto server 101 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 101 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0076] For some application software that provides audio source recognition functions such as music recognition, the client of the application software can be installed on the terminal 102. The client can collect the audio segments provided by the user based on the user's audio source recognition request and send the audio segments to the server 101. The server 101 can process the received audio segments to obtain the audio source corresponding to the audio segments and add the audio source to the music library in the data storage system.

[0077] The application scenarios mentioned above are only partial examples. Those skilled in the art can expand the applications based on the process of collecting music library audio. This disclosure does not impose specific limitations on these applications.

[0078] Exemplary methods

[0079] The following is combined Figure 1 Application scenarios, refer to Figures 2 to 8This description outlines a method for recording music library audio according to exemplary embodiments of this disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the embodiments of this disclosure are not limited in any way. Rather, the embodiments of this disclosure can be applied to any applicable scenario.

[0080] Figure 2 This is a flowchart illustrating a method for recording music in a music library according to an embodiment of this disclosure. Figure 2 As shown, the method may include:

[0081] Step S201: Obtain multiple first audio segments that failed to be recognized within the first duration.

[0082] The first duration can be any duration, such as one hour or one day.

[0083] For example Figure 1 In the application scenario shown, users can record audio clips using the software client on terminal 102. If an audio source corresponding to the audio clip is identified, the recognition is successful, and the client can display the audio source to the user. If no corresponding audio source is identified, the recognition fails. For example, if a user hears a song but does not know its name, they can record an audio clip containing part of the song's sound using the software client on terminal 102. The software client sends the audio clip to the software's backend server. If a song corresponding to the audio clip is matched and identified in the backend music library, the song can be returned to the client for display to the user. If the song is not included in the backend music library, the recognition failure result is returned to the client for display to the user.

[0084] Optionally, terminal 102 may send multiple audio segments that the software client failed to recognize within a first time period to server 101. These audio segments may be recorded or uploaded by different users through their respective clients.

[0085] Step S202: For each first audio segment, retrieve the recognition result that matches the first audio segment from among the multiple second audio segments that failed to recognize audio within the second duration.

[0086] The recognition result includes at least one second audio segment, and the first duration is shorter than the second duration.

[0087] Optionally, based on the similarity between the first audio segment and the second audio segment, a second audio segment that meets the preset similarity condition with the first audio segment can be retrieved from multiple second audio segments.

[0088] The following example illustrates the process of retrieving the recognition result that matches the first audio segment: Figure 3aAs shown in this example, the first audio segments that failed to be identified within the first duration include first audio segment A, first audio segment B, and first audio segment C. The second audio segments that failed to be identified within the second duration include second audio segment a, second audio segment b, and second audio segment c. If the similarity between first audio segment A and second audio segment a is greater than or equal to a preset threshold, and the similarity between first audio segment A and second audio segment b is also greater than or equal to a preset threshold, and the similarity between first audio segment A and second audio segment c is less than a preset threshold, then the identification result of first audio segment A includes second audio segment a and second audio segment b. Correspondingly, if the similarity between first audio segment B and second audio segment b is greater than or equal to a preset threshold, then the identification result of first audio segment B includes second audio segment b.

[0089] It is understandable that the second audio segment could be obtained in the same way as the first audio segment, which will not be elaborated here.

[0090] Step S203: Aggregate and mine the recognition results corresponding to multiple first audio segments respectively, and perform audio recognition based on the aggregation and mining results, and record the recognized audio into the music library.

[0091] The results of the aggregation mining include at least one second audio segment.

[0092] Optionally, the recognition results corresponding to each of the multiple first audio segments are processed by a clustering algorithm to mine similar audio features and identify the audio source based on the mining results. If the audio source corresponding to the audio segment is successfully identified, the audio source is recorded into the music library in the background.

[0093] It should be noted that the audio source in this disclosure can be a complete audio file containing audio segments. For example, if a user records an audio segment of a song, the audio source can be identified as the song itself based on that audio segment.

[0094] In some possible implementations, aggregating and mining the recognition results corresponding to multiple first audio segments can include: merging the recognition results corresponding to each first audio segment based on the similarity between different first audio segments, with the merged result including all recognition results of the first audio segments before merging. For example, refer to... Figure 3a In the example shown, if the similarity between the first audio segment A and the first audio segment B is greater than or equal to a preset similarity threshold, the recognition results of the first audio segment A and the first audio segment B can be merged. The merged recognition result includes the second audio segment a and the second audio segment b. The similarity can be cosine similarity.

[0095] In several possible implementations, audio recognition based on the results of aggregate mining can employ various methods to attempt to identify the audio source. For example, a different method than the one used to identify previously failed audio segments within the aggregate mining results can be employed. For instance, audio segments that failed to be identified within the first and second durations were identified using a music library matching method; however, for audio segments obtained from aggregate mining, methods such as manual listening can be used for further identification. Alternatively, the audio segments obtained from aggregate mining can be spliced ​​and combined, and then identified using the same or different methods as when previous audio recognition failed. For instance, audio segments that failed to be identified within the first and second durations were identified using manual listening; however, the audio segments from the aggregate mining results can be spliced ​​and combined and then manually listened to again to identify the song source.

[0096] It should be understood that the method for collecting music library audio in this disclosure can be executed by a computer device that can obtain manually input listening results to realize the music library audio collection process.

[0097] In some possible implementations, audio recognition based on the results of aggregate mining can include: after merging the recognition results of multiple first audio segments, the audio segments in the merged recognition result can be processed through audio recognition methods such as manual listening to obtain the recognized audio. For example, referring to... Figure 3a The example shown allows the merged recognition result to be given to experienced music recognition staff. These staff can listen to the second audio segment in the recognition result to obtain the corresponding complete audio, which can then be added to the music library. It should be understood that the complete audio obtained in this example is the audio source corresponding to both the first audio segment A and the first audio segment B. Using the method in this example, the identification of the audio source for the first audio segment A and the first audio segment B only requires selecting audio segments from the merged recognition result for identification, eliminating the need to separately identify the audio source for each segment. This reduces the workload caused by repeatedly identifying audio sources and improves recognition efficiency.

[0098] In some possible implementations, for the merged recognition results, a portion of the second audio segment can be randomly sampled for audio recognition.

[0099] In the above embodiments, by acquiring multiple first audio segments that failed audio recognition within a first duration, recognition results matching the first audio segments can be retrieved from multiple second audio segments that failed audio recognition within a second duration. Furthermore, based on the recognition results, aggregation mining can be performed before audio recognition, and the recognized audio can be entered into the music library. The music library audio collection method of this disclosure can perform aggregation mining based on first audio segments, eliminating the need to recognize all audio segments individually. By performing audio recognition on the aggregated and mined audio segments, the audio source of the corresponding audio segment can be quickly obtained and entered into the music library, thereby reducing the workload in the audio collection process and improving the efficiency of music library audio collection.

[0100] In one embodiment, such as Figure 4 As shown, the process of aggregating and mining the recognition results corresponding to multiple first audio segments, and then performing audio recognition based on the results of the aggregation and mining, can include:

[0101] Step S401: Merge the recognition results containing the same second audio segment to obtain the target audio segment set.

[0102] The target audio segment set includes all the second audio segments in the recognition results corresponding to at least two first audio segments.

[0103] Optionally, for the recognition results corresponding to multiple first audio segments, it can be determined whether to merge the recognition results based on whether these recognition results contain the same second audio segment. By merging the recognition results, a set of target audio segments can be obtained. For example, refer to... Figure 3a In the example shown, for the recognition results of the first audio segment A and the first audio segment B, it can be determined that both recognition results contain the same second audio segment b. Therefore, the two recognition results can be merged, and the merged recognition result contains the second audio segment a and the second audio segment b.

[0104] Furthermore, in determining whether to merge the recognition results for multiple first audio segments based on whether these recognition results contain the same second audio segment, the process can also be further determined based on whether the number of these identical second audio segments is greater than or equal to a preset merging threshold. If multiple recognition results contain the same second audio segment and the number of identical second audio segments is greater than or equal to the merging threshold, then the recognition results containing these second audio segments are merged. For example, refer to... Figure 5In the example shown, the recognition results of the first audio segment D include the second audio segment a, the second audio segment b, the second audio segment c, and the second audio segment d. The recognition results of the first audio segment E include the second audio segment a, the second audio segment b, and the second audio segment d. The two recognition results contain the same second audio segments: second audio segment a, second audio segment b, and second audio segment d. That is, the number of the same second audio segments in the two recognition results is 3. If the preset merging threshold is 3, the two recognition results can be merged.

[0105] In some possible implementations, a unique identifier can be assigned to each second audio segment, and the merging of recognition results can be determined based on whether the recognition results of different first audio segments contain second audio segments with the same identifier.

[0106] It should be understood that the above-mentioned merging of recognition results can be achieved by merging all recognition results that meet the conditions at once, or by merging some recognition results first, and then merging the merged recognition results with other recognition results. This disclosure does not limit the merging method of recognition results. Regardless of the merging method adopted, when the recognition results can no longer be merged, the obtained merged recognition results can be used as a target audio segment set, which includes the second audio segment from the merged recognition results.

[0107] Step S402: Based on the second audio segment in the target audio segment set, perform audio recognition using a preset audio recognition method to obtain the recognized audio.

[0108] The preset audio recognition method includes: performing audio recognition on at least one audio segment in the audio segment set through manual listening; or, splicing together at least two second audio segments in the target audio segment set, and using the spliced ​​audio segment to match the corresponding audio in the music library.

[0109] In some possible implementations, after obtaining the target audio segment set, the target audio segment set can be handed over to staff with rich music recognition experience for listening and identification. The staff can listen to the second audio segment in the target audio segment set and obtain the audio source of the corresponding second audio segment to be entered into the music library.

[0110] In some possible implementations, the second audio segment in the target audio segment set can be concatenated and combined. The concatenated audio segment can then be used to match and identify the audio source in the background music library. For example, the target audio segment set contains a second audio segment a and a second audio segment b. The second audio segment a contains the first and second lines of a song, and the second audio segment b contains the second and third lines of the same song. The audio segment concatenated and combined from the second audio segments a and b can contain the first, second, and third lines of the song. Using this concatenated and combined audio segment to match the audio source in the background music library can improve the success rate of audio source identification.

[0111] In some possible implementations, an artificial intelligence model can be used to identify the second audio segment in the target audio segment set, thereby obtaining the audio source of the corresponding second audio segment. This artificial intelligence model can be trained based on natural language processing algorithms such as speech recognition, and can convert the input audio into text content and match the corresponding audio source based on the text content.

[0112] In one embodiment, merging recognition results containing the same second audio segment to obtain a target audio segment set may include:

[0113] The first step is to determine whether any two recognition results corresponding to multiple first audio segments meet the merging conditions, and then merge the recognition results that meet the merging conditions. The merged recognition result includes all the second audio segments in the recognition results before merging.

[0114] The second step is to merge the merged recognition results and other recognition results that meet the merging conditions with the merged recognition results until there are no more recognition results that meet the merging conditions, thus obtaining the target audio segment set.

[0115] The merging condition is that the number of second audio segments in the intersection of the two recognition results is greater than or equal to a preset first threshold.

[0116] For example, refer to Figure 3b Example shown:

[0117] The first step is to determine whether the recognition results of the first audio segment A and the first audio segment B meet the merging conditions. First, the intersection of the second audio segments in the two recognition results is taken. The intersection includes the second audio segment b. If the preset first threshold in the merging conditions is 1, then the two recognition results meet the merging conditions if the number of second audio segments in the intersection is greater than or equal to 1. The recognition results of the first audio segment A and the first audio segment B are then merged. The merged recognition result includes the second audio segment a and the second audio segment b.

[0118] The second step is to determine whether the merged recognition result meets the merging condition with the recognition result of the first audio segment C. If the recognition result of C contains the second audio segment b and the second audio segment c, then the result of the second intersection of the merged recognition result and the recognition result of C contains the second audio segment b. If the number of the second audio segments in the second intersection is greater than or equal to 1, it is determined that the merging condition is met. The merged recognition result and the recognition result of C are merged again. The recognition result after the second merging contains the second audio segment a, the second audio segment b, and the second audio segment c.

[0119] It should be noted that, for any merged recognition result, if there are no other recognition results that can meet the merging conditions with the merged recognition result, then the merged recognition result can be determined as a target audio segment set.

[0120] In the above embodiments, by merging the recognition results corresponding to different first audio segments, not only can the success rate of audio source identification for the first audio segment be improved, but the workload of audio source identification for the first audio segment can also be reduced, thereby improving the efficiency of music library inclusion. For example, first audio segments A, B, and C may be segments obtained from recordings of the same song, each segment extracting a portion of the song's sound content. Originally, it would be necessary to identify the audio source song for each of these three segments, and if the effective sound in a certain first audio segment is too small (e.g., too much noise, poor sound quality, etc.), the recognition success rate will be low. However, through the method in the above embodiments, the merged recognition result contains more audio segments, and the audio segments in the same recognition result correspond to the same audio source, thus making it possible to identify more effective sounds from the same audio source, which can improve the audio source identification success rate. Furthermore, audio recognition can be performed only on the merged recognition result to obtain the audio to be included in the music library, without having to identify multiple first audio segments one by one, thereby reducing the workload of audio source identification.

[0121] In another embodiment, audio recognition is performed on the second audio segment in the target audio segment set using a preset audio recognition method to obtain the recognized audio. This may include: filtering a set of target audio segments to be recognized from multiple target audio segment sets, the filtering including a first filtering and / or a second filtering; and performing audio recognition on the audio segments in the set of target audio segments to be recognized using a preset audio recognition method to obtain the recognized audio.

[0122] The first filtering process includes: arranging multiple target audio segment sets in descending order of the number of audio segments they contain, and retaining the top K target audio segment sets as the first-filtered target audio segment set, where K is a positive integer. The second filtering process includes: determining the number of audio segments in any target audio segment set from the multiple target audio segment sets; if the number of audio segments in that target audio segment set is greater than or equal to a preset second threshold, then retaining that target audio segment set as the second-filtered target audio segment set.

[0123] In some possible implementations, if multiple sets of target audio segments are obtained, a first screening can be performed on these sets to obtain a combination of target audio segments to be identified. For example, in one example, after merging the recognition results, five sets of target audio segments are obtained, each containing several second audio segments. These sets of target audio segments are arranged in descending order of the number of second audio segments they contain, and the top three sets of the arranged sets are used for audio recognition.

[0124] In some possible implementations, if multiple sets of target audio segments are obtained, a second filtering can be performed on these sets to obtain a combination of target audio segments to be identified. For example, in another example, after merging the recognition results, seven sets of target audio segments are obtained. If the preset second threshold is 5, and only four of these seven sets contain a number of second audio segments greater than or equal to 5, then these four sets of target audio segments are used for audio recognition.

[0125] In the above embodiments, by filtering the target audio segment set, a set of target audio segments to be identified with a large number of audio segments can be obtained. It should be understood that each target audio segment set can correspond to an audio source, and each audio segment can correspond to one audio source identification request. The more audio segments a target audio segment set contains, the greater the user demand for identification of the audio source corresponding to that target audio segment set. This method can filter out audio source identification tasks with more urgent user needs and prioritize identification and audio recording for the corresponding target audio segment sets to address the large number of newly added audio source identification demands in a short period. Furthermore, by controlling the size of the first duration, the problem of music library audio recording that needs to be processed more urgently within a specified time can be addressed.

[0126] In yet another embodiment, such as Figure 6 As shown, retrieving a recognition result that matches the first audio segment from multiple second audio segments that failed to be recognized within the second duration can include:

[0127] Step S601: Extract the audio fingerprint of the first audio segment.

[0128] Optionally, after obtaining the first audio segment that failed to be recognized within the first duration, a short-time Fourier transform can be performed on the first audio segment to obtain its spectrogram. The energy maxima in the spectrogram can then be used as feature points to construct an audio fingerprint. For example, the audio fingerprint of the first audio segment can be generated based on local peak points in its spectrogram. The method for determining local peak points can include: if the value of a point in the spectrogram is greater than that of all surrounding points (such as a surrounding rectangular area), then that point is a local peak point.

[0129] Step S602: Among the multiple second audio segments that failed to be recognized within the second time period, the number of audio fingerprint matches between the audio fingerprint of the first audio segment and the audio fingerprint of the second audio segment is determined based on the audio fingerprint of the first audio segment and the audio fingerprint of the second audio segment.

[0130] The audio fingerprint of the second audio segment can be obtained in the same way as the audio fingerprint of the first audio segment, which will not be elaborated here.

[0131] Optionally, both the first audio segment and the second audio segment contain multiple audio fingerprints. The number of identical audio fingerprints contained in a single first audio segment and a single second audio segment can be calculated, and this number can be used as the number of audio fingerprint matches between the first audio segment and the second audio segment.

[0132] In some possible implementations, after constructing the audio fingerprint of the second audio segment, the audio fingerprint can also be stored as an inverted index. An inverted index is an indexing method that searches for records based on attribute values.

[0133] Step S603: If the number of audio fingerprint hits is greater than or equal to a preset third threshold, generate the recognition result of the first audio segment based on the second audio segment.

[0134] Optionally, if the number of audio fingerprint matches between the first audio segment and the second audio segment is greater than or equal to a preset third threshold, the second audio segment can be added to the recognition result of the first audio segment. It should be noted that the recognition result of the first audio segment can consist of one or more second audio segments.

[0135] In the above embodiments, by extracting audio fingerprints, second audio segments that match the audio fingerprints of the first audio segment a number that reaches a third threshold can be matched to generate a recognition result for the first audio segment. In this way, the second audio segment in the recognition result can be used as material for identifying the audio source of the first audio segment. If the audio source cannot be successfully identified using the first audio segment, the audio source can still be identified using the second audio segment in the recognition result, and the audio source identified based on the second audio segment can be used as the audio source of the first audio segment, thereby improving the success rate of identifying the audio source of the first audio segment.

[0136] In yet another embodiment, such as Figure 7 As shown, audio recognition based on the results of aggregation mining can include:

[0137] Step S701: Generate first auxiliary information based on the second audio segment in the result of aggregation mining.

[0138] The first auxiliary information may include the pitch curve of the second audio segment and / or the text information obtained after speech recognition of the second audio segment.

[0139] Optionally, the result of the aggregation mining may include the target audio segment set in the above embodiments. After obtaining the target audio segment set, for the second audio segment in the target audio segment set, a pitch melody extraction tool can be used to process the second audio segment and extract the pitch curve of the second audio segment as the first auxiliary information.

[0140] In some possible implementations, for the second audio segment in the target audio segment set, the second audio segment can also be processed by technologies such as speech recognition to obtain text information such as lyrics recognized by speech as the first auxiliary information.

[0141] Step S702: Perform audio recognition on the second audio segment based on the first auxiliary information to obtain the recognized audio.

[0142] Optionally, after the set of target audio segments is given to staff for manual listening and recognition, when the staff listens to the second audio segment, the first auxiliary information corresponding to the second audio segment can be provided to the staff so that they can identify the audio source by combining it with text information such as pitch curves or lyrics, thereby improving the recognition success rate.

[0143] In yet another embodiment, such as Figure 8 As shown, audio recognition based on the results of aggregation mining can also include:

[0144] Step S801: Obtain the second auxiliary information.

[0145] The second auxiliary information can be the voice and / or text information entered by the user after failing to recognize the first audio segment.

[0146] Optionally, if the user fails to identify the first audio segment using the software, the software can obtain the user's input text information by providing a human-computer interaction interface on the client side. For example, if the user records the first audio segment using their mobile phone and tries to identify the audio source using the software's song recognition function, and the identification fails, an input box can be provided on the software interface so that the user can input second auxiliary information related to the first audio segment (such as the first audio segment possibly being sung by a certain singer).

[0147] In some possible implementations, secondary auxiliary information can also be obtained from user input, such as voice input.

[0148] In some possible implementations, the second auxiliary information may also be voice and / or text information entered by the user after the second audio segment fails to be recognized.

[0149] Step S802: Based on the second auxiliary information, perform audio recognition on the second audio segment in the result of aggregation mining to obtain the recognized audio.

[0150] In the above embodiments, by obtaining the second auxiliary information, the second auxiliary information can be used for audio source recognition, thereby improving the recognition success rate.

[0151] Exemplary media

[0152] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 9 The storage medium of the exemplary embodiments of this disclosure will be described.

[0153] refer to Figure 9 As shown, the storage medium 90 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0154] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0155] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0156] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0157] Exemplary device

[0158] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 10 The audio recording device for a music library according to an exemplary embodiment of the present disclosure is described in detail. It is used to implement the method in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0159] like Figure 10 As shown, the music library audio recording device 1000 may include:

[0160] The acquisition module 1001 is used to acquire multiple first audio segments that failed to be recognized within a first duration.

[0161] The retrieval module 1002 is configured to, for each first audio segment, retrieve a recognition result matching the first audio segment from among a plurality of second audio segments that failed to be recognized within a second duration, wherein the recognition result includes at least one second audio segment and the first duration is less than the second duration.

[0162] The mining and recognition module 1003 is used to aggregate and mine the recognition results corresponding to the plurality of first audio segments respectively, and to perform audio recognition based on the aggregate mining results, and to record the recognized audio into the music library. The aggregate mining results include at least one second audio segment.

[0163] In yet another embodiment of this disclosure, the mining and identification module 1003 may include:

[0164] The recognition result merging unit can be used to merge recognition results containing the same second audio segment to obtain a target audio segment set, wherein the target audio segment set contains all the second audio segments in the recognition results corresponding to at least two first audio segments.

[0165] The audio recognition unit can be used to perform audio recognition based on a second audio segment in the target audio segment set using a preset audio recognition method to obtain the recognized audio; wherein, the preset audio recognition method includes: performing audio recognition on at least one audio segment in the audio segment set through manual listening; or, splicing and combining at least two second audio segments in the target audio segment set, and using the spliced ​​and combined audio segment to match the corresponding audio in the music library.

[0166] In yet another embodiment of this disclosure, the identification result merging unit may include:

[0167] The first merging unit can be used to determine whether any two recognition results from the recognition results corresponding to the plurality of first audio segments meet the merging conditions, and merge the recognition results that meet the merging conditions. The merged recognition result includes all the second audio segments in the recognition results before merging.

[0168] The second merging unit can be used to merge the merged recognition result and other recognition results that meet the merging conditions with the merged recognition result until no recognition result meets the merging conditions, thereby obtaining a target audio segment set. The merging condition is that the number of second audio segments in the intersection of two recognition results is greater than or equal to a preset first threshold.

[0169] In yet another embodiment of this disclosure, the audio recognition unit may include:

[0170] The filtering unit can be used to filter out a set of target audio segments to be identified from a set of multiple target audio segments, wherein the filtering includes a first filtering and / or a second filtering.

[0171] The recognition unit can be used to perform audio recognition based on audio segments in the set of target audio segments to be recognized using a preset audio recognition method, and obtain the recognized audio. The first filtering includes: arranging the multiple target audio segment sets in descending order of the number of audio segments they contain, and retaining the first K target audio segment sets as the first-filtered target audio segment set, where K is a positive integer. The second filtering includes: determining the number of audio segments in any target audio segment set from the multiple target audio segment sets; if the number of audio segments in the target audio segment set is greater than or equal to a preset second threshold, then retaining the target audio segment set as the second-filtered target audio segment set.

[0172] In yet another embodiment of this disclosure, the retrieval module 1002 may include:

[0173] The fingerprint extraction unit can be used to extract the audio fingerprint of the first audio segment.

[0174] The fingerprint matching unit can be used to determine the number of audio fingerprint matches between the audio fingerprints of the first audio segment and the second audio segment in multiple second audio segments that fail to be recognized within a second time period, based on the audio fingerprints of the first audio segment and the second audio segment.

[0175] The recognition result generation unit can be used to generate the recognition result of the first audio segment based on the second audio segment when the number of audio fingerprint hits is greater than or equal to a preset third threshold.

[0176] In yet another embodiment of this disclosure, the fingerprint extraction unit may include:

[0177] The audio transformation unit can be used to perform a short-time Fourier transform on the first audio segment to obtain the spectrogram of the first audio segment.

[0178] The fingerprint generation unit can be used to generate an audio fingerprint of the first audio segment based on local peak points in the spectrogram of the first audio segment.

[0179] In yet another embodiment of this disclosure, the mining and identification module 1003 may include:

[0180] The first auxiliary acquisition unit can be used to generate first auxiliary information based on the second audio segment in the result of the aggregation mining.

[0181] The first auxiliary recognition unit can be used to perform audio recognition on the second audio segment according to the first auxiliary information to obtain the recognized audio. The first auxiliary information includes the pitch curve of the second audio segment and / or the text information obtained after speech recognition of the second audio segment.

[0182] In yet another embodiment of this disclosure, the mining and identification module 1003 may further include:

[0183] The second auxiliary acquisition unit can be used to acquire second auxiliary information, which is the voice and / or text information input by the user after failing to recognize the first audio segment.

[0184] The second auxiliary recognition unit can be used to perform audio recognition on the second audio segment in the result of the aggregation mining based on the second auxiliary information, so as to obtain the recognized audio.

[0185] Exemplary computing device

[0186] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 11 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0187] Figure 11 The computing device 110 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0188] like Figure 11 As shown, the computing device 110 is presented in the form of a general-purpose computing device. The components of the computing device 110 may include, but are not limited to: at least one processing unit 1101, at least one storage unit 1102, and a bus 1103 connecting different system components (including the processing unit 1101 and the storage unit 1102). The at least one storage unit 1102 stores computer-executable instructions; the at least one processing unit 1101 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0189] Bus 1103 includes a data bus, a control bus, and an address bus.

[0190] Storage unit 1102 may include readable media in the form of volatile memory, such as random access memory (RAM) 11021 and / or cache memory 11022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 11023.

[0191] Storage unit 1102 may also include a program / utility 11025 having a set (at least one) of program modules 11024, such program modules 11024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0192] The computing device 110 can also communicate with one or more external devices 1104 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 1105. Furthermore, the computing device 110 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1106. Figure 11 As shown, network adapter 1106 communicates with other modules of computing device 110 via bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 110, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0193] It should be noted that although several units / modules or sub-units / modules of the music library audio recording device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0194] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0195] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for collecting audio data from a music library, characterized in that, include: Retrieve multiple first audio segments that failed to be recognized within the first duration; For each first audio segment, among multiple second audio segments that fail to be recognized within a second duration, a recognition result matching the first audio segment is retrieved, wherein the recognition result includes at least one second audio segment, and the first duration is less than the second duration; The recognition results corresponding to the multiple first audio segments are aggregated and mined, and audio recognition is performed based on the aggregated mining results. The recognized audio is then recorded into the music library. The aggregated mining results include at least one second audio segment.

2. The method for collecting music library audio according to claim 1, characterized in that, The step of aggregating and mining the recognition results corresponding to the plurality of first audio segments respectively, and performing audio recognition based on the aggregation and mining results, includes: The recognition results containing the same second audio segment are merged to obtain a target audio segment set, which contains all the second audio segments in the recognition results corresponding to at least two first audio segments respectively; Based on the second audio segment in the target audio segment set, audio recognition is performed using a preset audio recognition method to obtain the recognized audio; The preset audio recognition method includes: performing audio recognition on at least one audio segment in the audio segment set through manual listening; or, splicing together at least two second audio segments in the target audio segment set, and using the spliced ​​audio segment to match the corresponding audio in the music library.

3. The method for collecting music library audio according to claim 2, characterized in that, The step of merging the recognition results containing the same second audio segment to obtain a target audio segment set includes: From the recognition results corresponding to the plurality of first audio segments respectively, determine whether any two recognition results meet the merging condition, and merge the recognition results that meet the merging condition. The merged recognition result includes all the second audio segments in the recognition results before merging. The merged recognition result and other recognition results that meet the merging conditions are merged until no recognition result meets the merging conditions, thus obtaining a set of target audio segments; The merging condition is that the number of second audio segments in the intersection of the two recognition results is greater than or equal to a preset first threshold.

4. The method for collecting music library audio according to claim 2 or 3, characterized in that, The audio recognition based on the second audio segment in the target audio segment set is performed using a preset audio recognition method to obtain the recognized audio, including: The set of target audio segments to be identified is selected from multiple sets of target audio segments, wherein the selection includes a first selection and / or a second selection; Based on the audio segments in the set of target audio segments to be identified, audio recognition is performed using a preset audio recognition method to obtain the identified audio; The first filtering includes: arranging the multiple target audio segment sets in descending order of the number of audio segments they contain, and retaining the first K target audio segment sets after the first filtering as the target audio segment set, where K is a positive integer; The second filtering includes: determining the number of audio segments contained in any target audio segment set among the plurality of target audio segment sets; if the number of audio segments contained in the target audio segment set is greater than or equal to a preset second threshold, then the target audio segment set is retained as the target audio segment set after the second filtering.

5. The method for recording music library audio according to any one of claims 1 to 3, characterized in that, Among the multiple second audio segments that failed to be recognized within the second duration, retrieving the recognition result that matches the first audio segment includes: Extract the audio fingerprint of the first audio segment; In the second time period, among the multiple second audio segments that failed to be recognized, the number of audio fingerprint matches between the audio fingerprint of the first audio segment and the audio fingerprint of the second audio segment is determined based on the audio fingerprint of the first audio segment and the audio fingerprint of the second audio segment. If the number of audio fingerprint hits is greater than or equal to a preset third threshold, the recognition result of the first audio segment is generated based on the second audio segment.

6. The method for collecting music library audio according to claim 5, characterized in that, Extracting the audio fingerprint of the first audio segment includes: Perform a short-time Fourier transform on the first audio segment to obtain the spectrogram of the first audio segment; An audio fingerprint of the first audio segment is generated based on the local peak points in the spectrogram of the first audio segment.

7. The method for recording music library audio according to any one of claims 1 to 3, characterized in that, The audio recognition based on the results of aggregation mining includes: First auxiliary information is generated based on the second audio segment in the results of the aggregated mining; The second audio segment is subjected to audio recognition based on the first auxiliary information to obtain the recognized audio. The first auxiliary information includes the pitch curve of the second audio segment and / or the text information obtained after speech recognition of the second audio segment.

8. The method for recording music library audio according to any one of claims 1 to 3, characterized in that, The audio recognition based on the results of aggregation mining includes: Obtain second auxiliary information, which is the voice and / or text information input by the user after failing to recognize the second audio segment; Based on the second auxiliary information, the second audio segment in the result of the aggregated mining is subjected to audio recognition to obtain the recognized audio.

9. A music library audio recording device, characterized in that, include: The acquisition module is used to acquire multiple first audio segments that failed to be recognized within the first duration. The retrieval module is configured to, for each first audio segment, retrieve a recognition result matching the first audio segment from among a plurality of second audio segments that failed to be recognized within a second duration, wherein the recognition result includes at least one second audio segment and the first duration is less than the second duration; The mining and recognition module is used to aggregate and mine the recognition results corresponding to the multiple first audio segments respectively, and to perform audio recognition based on the aggregate mining results, and to record the recognized audio into the music library. The aggregate mining results include at least one second audio segment.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 8.

11. A computing device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor to cause the computing device to perform the method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images