A method, device, electronic device and storage medium for generating an audio library

Abstract

The present application relates to the field of computer technology, and in particular to the field of artificial intelligence technology, and provides a method, device, electronic device and storage medium for generating an audio library to improve the timeliness of the audio library, wherein the method comprises: after the audio to be identified fails to match the first audio library, the audio to be identified is matched with the second audio library, wherein the second audio library is established based on each second audio that fails to match the first audio library, and if there is a target second audio whose cumulative number of successful matches reaches a preset threshold value among the successfully matched second audios, the target second audio is transferred to the first audio library. By storing the audio to be identified that fails to match the first audio library in the second audio library, and transferring it to the first audio library when the cumulative number of matches reaches the preset threshold value, the timeliness of the audio library is improved, thereby achieving accurate audio recommendation.

Description

Background Art

[0002] With the rapid development of user-generated content (UGC) multimedia content, audio content (abbreviated as audio) has become an important part of multimedia content, such as background music (BGM); the use of audio content plays an important role in multimedia content, such as the classification, recommendation, and re-creation of multimedia content.

[0003] In the related art, audio content is used by pre-building an audio library, such as audio content recognition, etc. However, in this method, audio content that does not exist in the audio library cannot be recognized in time, especially when a large amount of multimedia content is uploaded at the same time, it is difficult to recognize the newly uploaded audio content in time, resulting in poor timeliness of the audio library. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and storage medium for generating an audio library, so as to improve the timeliness of the audio library and ensure timely identification of audio content that does not exist in the audio library.

[0005] In a first aspect, an embodiment of the present application provides a method for generating an audio library, comprising:

[0006] Matching the audio to be identified with a preset first audio library, and if the match fails, matching the audio to be identified with a preset second audio library to obtain a target matching result; wherein the second audio library is established based on each second audio that fails to match the first audio library;

[0007] If, based on the target matching result, it is determined that the audio to be recognized successfully matches at least one second audio in the second audio library, and among the at least one second audio, there is a target second audio whose cumulative number of successful matches reaches a preset threshold, then the target second audio is obtained;

[0008] The target second audio is transferred to the first audio library as a new first audio.

[0009] In a second aspect, an embodiment of the present application provides a device for generating an audio library, including:

[0010] A matching unit, configured to match the audio to be identified with a preset first audio library, and if the match fails, to match the audio to be identified with a preset second audio library to obtain a target matching result; wherein the second audio library is established based on each second audio that fails to match the first audio library;

[0011] a determination unit, configured to obtain the target second audio if it is determined based on the target matching result that the audio to be recognized matches the at least one second audio in the second audio library successfully, and there is a target second audio whose cumulative number of successful matches reaches a preset threshold value in the at least one second audio;

[0012] The transfer unit is used to transfer the target second audio as a new first audio to the first audio library.

[0013] Optionally, when determining that the audio to be recognized meets a preset audio detection condition, the determining unit is used to:

[0014] Input the audio to be recognized into a trained target audio detection model to obtain a detection prediction value; wherein the target audio detection model is obtained by training the audio detection model to be trained based on an audio annotation dataset;

[0015] When it is determined that the detection prediction value reaches a preset prediction threshold value, it is determined that the audio to be identified meets the audio detection condition.

[0016] Optionally, before matching the audio to be recognized with a preset first audio library, the matching unit is further configured to:

[0017] Acquire multimedia content to be identified, and extract the audio to be identified from the multimedia content;

[0018] Then, after transferring the second audio as a new first audio to the first audio library, the transferring unit is further used for:

[0019] The new first audio is used as the audio recognition result of the multimedia content, and the multimedia identifier of the multimedia content is recorded corresponding to the new first audio.

[0020] Optionally, the device further comprises an associating unit, and the associating unit is used to:

[0021] Based on every two first audios in the first audio library, generate a corresponding first audio group;

[0022] For each first audio group obtained, perform the following operations respectively:

[0023] Determine two first audios included in one of the first audio groups;

[0024] Acquire a set of multimedia identifiers corresponding to each of the two first audios; wherein each multimedia identifier represents a multimedia content that successfully matches the corresponding first audio;

[0025] If the number of repeated multimedia identifiers in the two acquired multimedia identifier sets reaches a preset number threshold, a new audio identifier is generated corresponding to the two first audio associations.

[0026] Optionally, the device further comprises a recommendation unit, wherein the recommendation unit is configured to:

[0027] In response to an input operation triggered in the client, based on the respective corresponding multimedia identification sets of the respective first audios contained in the first audio library, a recommended audio set is determined from the respective first audios; wherein each multimedia identification represents a multimedia content that successfully matches the corresponding first audio;

[0028] The recommended audio set is presented in the client.

[0029] Optionally, when determining the recommended audio set from each first audio based on the multimedia identification set corresponding to each first audio contained in the first audio library, the recommendation unit is specifically configured to:

[0030] Acquire a set of multimedia identifiers corresponding to each first audio contained in the first audio library, and obtain an evaluation value corresponding to each first audio based on the acquired set of multimedia identifiers; wherein each evaluation value is used to characterize the usage status of the corresponding first audio;

[0031] Based on the evaluation values ​​corresponding to the respective first audios, the respective first audios are sorted to obtain a target sequence, and a set number of first audios are selected in sequence from the target sequence as a recommended audio set.

[0032] Optionally, when obtaining the evaluation values ​​corresponding to the first audios based on the obtained multimedia identification sets, the recommendation unit is specifically configured to:

[0033] Based on the acquired multimedia identifier set corresponding to one of the first audios, acquiring the interaction status information of each corresponding multimedia content;

[0034] Based on the interaction status information of each multimedia content and the preset mapping relationship between the interaction status information and the weight, obtaining the weight corresponding to each multimedia content;

[0035] Based on the weights corresponding to the multimedia contents, an evaluation value of the first audio is obtained.

[0036] Optionally, when matching the audio to be recognized with a preset first audio library, the matching unit is specifically used to:

[0037] Based on a preset audio fingerprint extraction algorithm, extract the audio fingerprint of the audio to be identified to obtain an audio fingerprint to be identified corresponding to the audio to be identified; wherein the audio fingerprint to be identified is used to characterize the audio features corresponding to the audio to be identified;

[0038] Based on the audio fingerprint extraction algorithm, extract audio fingerprints from each first audio contained in the first audio library to obtain first audio fingerprints corresponding to each first audio; wherein each first audio fingerprint is used to characterize audio features of a corresponding first audio;

[0039] Calculating the similarities between the audio fingerprint to be identified and each of the first audio fingerprints obtained;

[0040] If there is at least one first audio fingerprint among the first audio fingerprints whose similarity with the audio fingerprint to be identified reaches a preset similarity threshold, it is determined that the audio to be identified is successfully matched with the first audio library.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned audio library generation method.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the above-mentioned audio library generation method.

[0043] In an embodiment of the present application, after the audio to be identified fails to be matched with the first audio library, the audio to be identified is matched with the second audio library, wherein the second audio library is established based on each second audio that fails to match with the first audio library, and if among the successfully matched second audios, there is a target second audio whose cumulative number of successful matches reaches a preset threshold value, then the target second audio is transferred to the first audio library.

[0044] In this way, since the audio to be identified can be stored in the second audio library as the second audio after failing to match the first audio library, the matching success rate of the audio to be identified is improved. In addition, by transferring the target second audio to the first audio library, the timeliness of the first audio library is improved, ensuring that the audio that does not exist in the first audio library is identified in time, thereby further improving the matching success rate of the audio to be identified.

[0045] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1A A schematic diagram of a possible application scenario provided in an embodiment of the present application;

[0048] Figure 1B A schematic diagram of another possible application scenario provided in an embodiment of the present application;

[0049] Figure 1C A schematic diagram of a blockchain provided in an embodiment of the present application;

[0050] Figure 1D A flowchart of a block generation method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a flow chart of a method for generating an audio library provided in an embodiment of the present application;

[0052] Figure 3A A schematic diagram of a flow chart of a method for matching an audio to be recognized with a first audio library provided in an embodiment of the present application;

[0053] Figure 3B A logical schematic diagram of a matching process between an audio to be recognized and a first audio library provided in an embodiment of the present application;

[0054] Figure 4 A logical schematic diagram of a situation in which an audio to be recognized is successfully matched with a second audio library provided in an embodiment of the present application;

[0055] Figure 5 A logical schematic diagram of a situation in which a to-be-recognized audio fails to match a second audio library provided in an embodiment of the present application;

[0056] Figure 6 A logical schematic diagram of obtaining audio to be recognized provided in an embodiment of the present application;

[0057] Fig. 7A A schematic diagram of a multimedia identification set provided in an embodiment of the present application;

[0058] Figure 7B A schematic diagram of a first audio library provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of a flow chart of an audio recommendation method based on a first audio library provided in an embodiment of the present application;

[0060] Fig.9A A schematic diagram of an application operation interface provided in an embodiment of the present application;

[0061] Fig. 9B A schematic diagram of presenting a recommended audio set provided in an embodiment of the present application;

[0062] Fig.10 A schematic diagram of generating a popular sound effect library provided in an embodiment of the present application;

[0063] Fig.11 A schematic diagram of the structure of a detection device provided in an embodiment of the present application;

[0064] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0065] Fig.13 A schematic diagram of the hardware composition structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0067] The following is an introduction to some concepts involved in the embodiments of the present application.

[0068] 1. Multimedia content. The multimedia content in the embodiments of the present application includes but is not limited to content including two or more media, and the media includes but is not limited to text, data, images, animations, audio, etc.

[0069] 2. Audio content. The audio content in the embodiment of the present application includes but is not limited to background music, special effect sounds, etc., wherein the special effect sounds may be laughter, cheers, etc. For the convenience of description, the audio content is referred to as audio in the following text.

[0070] 3. Audio fingerprint: The audio corresponding to the audio content is transformed into a time-frequency spectrum, such as using fast Fourier transform (FFT) to transform, and an audio fingerprint representing the identity characteristics of the audio content is constructed based on the statistical characteristics of the time-frequency peak in the time-frequency spectrum. When the similarity of the audio fingerprints of two audio contents exceeds the similarity threshold, it can be determined that the two audio contents are the same, and the similarity of the audio fingerprints reflects the similarity of the two audio contents.

[0071] The embodiments of the present application relate to artificial intelligence (AI) and machine learning technology, and are designed based on voice technology and machine learning (ML) in artificial intelligence.

[0072] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0073] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0074] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0075] The embodiment of the present application adopts a machine learning audio detection model when determining that the audio to be identified meets the preset audio detection conditions. The method for training the audio detection model proposed in the embodiment of the present application can be divided into two parts, including a training part and an application part; wherein, the training part involves the technical field of machine learning. In the training part, the audio detection model is trained by the technology of machine learning, and the audio annotation data set given in the embodiment of the present application is used as a training data set to train the audio detection model. After the training data in the training data set is input into the audio detection model, the output result of the audio detection model is obtained, and the model parameters are continuously adjusted through the optimization algorithm in combination with the output result; the application part is used to use the audio detection model obtained in the training of the training part to detect the audio to be identified, and obtain the inspection prediction value of the audio to be identified. In addition, it should be noted that the audio detection model in the embodiment of the present application can be either online training or offline training, which is not specifically limited here.

[0076] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0077] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0078] Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time frame. It is a massive, high-growth, and diverse information asset that requires new processing models to have stronger decision-making power, insight discovery, and process optimization capabilities. With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special technologies to effectively process large amounts of data within a tolerable time frame. Technologies applicable to big data include large-scale parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0079] With the rapid development of UGC multimedia content, audio content has become an important part of multimedia content, such as BGM, special effects sound, etc. The use of audio content plays an important role in multimedia content, such as the classification, recommendation, and re-creation of multimedia content.

[0080] In the related art, audio content is used by pre-building an audio library, such as identifying audio content, and then recommending popular audio, multimedia content, and multimedia content classification based on the audio content recognition results. However, in this way, audio content that does not exist in the audio library cannot be identified in time, especially when a large amount of multimedia content is uploaded at the same time, it is difficult to identify the newly uploaded audio content in time, resulting in poor timeliness of the audio library.

[0081] Due to the poor timeliness of the audio library, the audio content that does not exist in the audio library cannot be identified in time. In the embodiment of the present application, a popular audio library and a temporary audio library are constructed. When the audio to be identified fails to match the popular audio library, the audio to be identified is used as temporary audio and stored in the temporary audio library. When the cumulative number of successful matches of the temporary audio reaches a set threshold value, the temporary audio is used as hot audio and stored in the hot audio library.

[0082] In this way, since the audio to be identified can be stored in the temporary music library as temporary audio after the audio to be identified fails to match the popular audio library, the matching success rate of the audio to be identified is improved. In addition, by storing temporary audio whose cumulative matching success times reach the set threshold value in the popular audio library, the timeliness of the popular audio library is improved, ensuring that the audio that does not exist in the popular audio library is identified in time, thereby further improving the matching success rate of the audio to be identified.

[0083] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0084] See also Figure 1A As shown, it is a schematic diagram of a possible application scenario in an embodiment of the present application. The application scenario includes a terminal device 110, a server 120 and a data storage node 130. The terminal device 110, the server 120 and the data storage node 130 communicate with each other through a communication network.

[0085] In a possible implementation, the communication network is a wired network or a wireless network. The terminal device 110, the server 120, and the data storage node 130 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0086] The user logs in to the application operation interface through the terminal device 110, and the terminal device 110 uploads multimedia content to the multimedia service system deployed on the server 120 by responding to the operation triggered by the user in the application operation interface, so that the server 120 generates an audio library based on the multimedia content uploaded by the terminal device 110. For example, the server obtains popular audio and establishes a popular audio library based on the multimedia content uploaded by the terminal device 110, and obtains temporary audio and establishes a temporary audio library. Exemplarily, after the terminal device 110 responds to the user operation, it can also receive and present the recommended audio collection returned by the server 120.

[0087] In the embodiment of the present application, the application can be a social software, such as an instant messaging software, a short video software, or a small program, a web page, etc., which is not specifically limited here. Among them, the terminal device 110 needs to be installed with an application, where the application can be software, or a web page, a small program, etc., and the server 120 is a server corresponding to the software or web page, small program, etc.

[0088] In the embodiment of the present application, the terminal device 110 is an electronic device used by the user, which may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. Each terminal device 110 is connected to the server 120 via a wireless network. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0089] The data storage node 130 communicates with the server 120 via a communication network. The data storage node 130 is used to store the popular audio obtained by the server 120 and to store the temporary audio obtained by the server 120 .

[0090] In a possible implementation, the data storage node 130 may be in the form of a database for data storage. The popular audio and temporary audio may be stored in the same database or in different databases, which is not limited in the present application.

[0091] In another possible implementation, see Figure 1B As shown, the data storage node 130 may be implemented in the form of a data sharing system 140 for data storage.

[0092] The data sharing system 140 refers to a system for sharing data between nodes, and the data sharing system may include multiple nodes 141, and the multiple nodes 141 may refer to each client in the data sharing system. Each node 141 can receive input information when performing normal work, and maintain the shared data in the data sharing system based on the received input information. In order to ensure the information intercommunication within the data sharing system, there may be an information connection between each node in the data sharing system, and information can be transmitted between nodes through the above information connection. For example, when any node in the data sharing system receives input information, other nodes in the data sharing system obtain the input information according to the consensus algorithm, and store the input information as data in the shared data, so that the data stored on all nodes in the data sharing system are consistent.

[0093] Each node in the data sharing system has a node identifier corresponding to it, and each node in the data sharing system can store the node identifiers of other nodes in the data sharing system, so that the generated blocks can be broadcast to other nodes in the data sharing system according to the node identifiers of other nodes. A node identifier list as shown in the following table can be maintained in each node, and the node name and node identifier are stored in the node identifier list accordingly. Among them, the node identifier can be an Internet Protocol (IP) address and any other information that can be used to identify the node. Table 1 only takes the IP address as an example for illustration.

[0094] Table 1 Node identification list

[0095] Node Name Node ID Node 1 117.114.151.174 Node 2 117.116.189.145 …… …… Node N 119.123.789.258

[0096] Each node in the data sharing system stores the same blockchain. The blockchain consists of multiple blocks, see Figure 1C The blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores the input information characteristic value, version number, timestamp and difficulty value, and the block body stores the input information; the next block of the genesis block takes the genesis block as the parent block, and the next block also includes a block header and a block body. The block header stores the input information characteristic value of the current block, the block header characteristic value, version number, timestamp and difficulty value of the parent block, and so on, so that the block data stored in each block in the blockchain is associated with the block data stored in the parent block, ensuring the security of the input information in the block.

[0097] When generating each block in the blockchain, see Figure 1DWhen the node where the blockchain is located receives the input information, it verifies the input information. After the verification is completed, the input information is stored in the memory pool and the hash tree used to record the input information is updated. After that, the update timestamp is updated to the time when the input information is received, and different random numbers are tried to calculate the eigenvalue multiple times so that the calculated eigenvalue can satisfy the following formula:

[0098] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x)) <TARGET

[0099] Among them, SHA256 is the eigenvalue algorithm used to calculate the eigenvalue; the version number (version) is the version information of the relevant block protocol in the blockchain; prev_hash is the block header eigenvalue of the parent block of the current block; merkle_root is the eigenvalue of the input information; ntime is the update time of the update timestamp; nbits is the current difficulty, which is a fixed value within a period of time and is determined again after exceeding the fixed time period; x is a random number; TARGET is the eigenvalue threshold, which can be determined based on nbits.

[0100] In this way, when the random number that satisfies the above formula is calculated, the information can be stored accordingly, the block header and block body can be generated, and the current block can be obtained. Subsequently, the node where the blockchain is located sends the newly generated block to other nodes in the data sharing system according to the node identification of other nodes in the data sharing system. Other nodes verify the newly generated block and add the newly generated block to the blockchain stored in them after the verification is completed.

[0101] See also Figure 2 As shown, it is a flow chart of a method for generating an audio library provided in an embodiment of the present application. The method for generating an audio library is applied to a device for generating an audio library. The device for generating an audio library may be a server 120 or a device deployed in the server 120. The specific implementation process of the method is as follows:

[0102] S201, the generating device matches the audio to be identified with a preset first audio library. If the match fails, the audio to be identified is matched with a preset second audio library to obtain a target matching result; wherein the second audio library is established based on each second audio that fails to match the first audio library.

[0103] Specifically, when executing S201, the following steps may be adopted but are not limited to:

[0104] S2011. The generating device matches the audio to be recognized with a preset first audio library to obtain a first matching result.

[0105] It should be noted that, in the embodiment of the present application, the first audio library may also be referred to as a popular audio library, and the second audio library may also be referred to as a temporary audio library.

[0106] In some embodiments, see Figure 3A As shown, in order to improve the matching efficiency, when executing S2011, the following steps can be adopted:

[0107] S20111. The generating device extracts the audio fingerprint of the audio to be identified based on a preset audio fingerprint extraction algorithm, and obtains the audio fingerprint to be identified corresponding to the audio to be identified.

[0108] Among them, the audio fingerprint to be identified is used to characterize the audio features corresponding to the audio to be identified. In the embodiment of the present application, the audio features include but are not limited to one or more of zero crossing rate, short-time energy, short-time average amplitude difference, spectrum diagram, short-time power spectrum density, spectrum entropy, fundamental frequency, resonance peak, etc. Taking the zero crossing rate as an example, the zero crossing rate refers to the number of times the sign of a signal changes in each frame, for example, the signal changes from a positive number to a negative number or from a negative number to a positive number.

[0109] In the embodiment of the present application, the audio fingerprint extraction algorithm may adopt but is not limited to an echoprint algorithm, a landmark algorithm, a chromaprint algorithm, and the like.

[0110] Taking BGM-A as an example, the generating device uses the echoprint algorithm to extract the audio fingerprint of BGM-A, and obtains the audio fingerprint to be identified corresponding to BGM-A, wherein the audio fingerprint to be identified represents that the zero-crossing rate of the first frame of BGM-A is 9, and the zero-crossing rate of the second frame is 10.

[0111] S20112. The generating device extracts audio fingerprints from each first audio contained in the first audio library based on an audio fingerprint extraction algorithm to obtain first audio fingerprints corresponding to each first audio.

[0112] Each first audio fingerprint is used to represent an audio feature of a corresponding first audio.

[0113] For example, see Figure 3B As shown, assuming that the first audio library contains BGM1, BGM2, and BGM3, the generating device extracts audio fingerprints of BGM1, BGM2, and BGM3 contained in the first audio library to obtain audio fingerprints corresponding to BGM1, BGM2, and BGM3 respectively.

[0114] S20113. The generating device calculates the similarity between the audio fingerprint to be identified and each of the obtained first audio fingerprints.

[0115] It should be noted that, in the embodiment of the present application, the similarity between the audio fingerprint to be identified and one of the first audio fingerprints is used to represent the number of frames containing the same audio fingerprint as the audio fingerprint to be identified and the first audio fingerprint.

[0116] For example, see Figure 3B As shown, the generating device calculates the similarity between the audio fingerprint to be identified and the audio fingerprints of BGM1, BGM2, and BGM3 respectively, wherein the value of similarity 1 between the audio fingerprint to be identified and the audio fingerprint of BGM1 is 50%, the value of similarity 2 between the audio fingerprint to be identified and the audio fingerprint of BGM2 is 80%, and the value of similarity 3 between the audio fingerprint to be identified and the audio fingerprint of BGM3 is 90%.

[0117] S20114. The generating device determines whether there is at least one first audio fingerprint among the first audio fingerprints whose similarity with the audio fingerprint to be identified reaches a preset similarity threshold. If so, execute S20115; otherwise, execute S20116.

[0118] S20115. The generating device obtains a first matching result, where the first matching result indicates that the audio to be identified successfully matches the preset first audio library.

[0119] For example, assuming that the preset similarity threshold is 85%, at this time, the similarity between the audio fingerprint to be identified and the audio fingerprints of BGM1 and BGM2 does not reach 85%, and the similarity between the audio fingerprint to be identified and the audio fingerprint of BGM3 reaches 85%. The generating device determines that among the audio fingerprints of BGM1, BGM2, and BGM3, there is a first audio fingerprint whose similarity with the audio fingerprint to be identified reaches the preset similarity threshold, and obtains a first matching result. The first matching result indicates that BGM-A is successfully matched with the first audio library.

[0120] S20116. The generating device obtains a first matching result, where the first matching result indicates that the audio to be identified fails to match the preset first audio library.

[0121] For example, assuming that the preset similarity threshold is 95%, at this time, the similarities between the audio fingerprint to be identified and the first audio fingerprints of BGM1, BGM2, and BGM3 do not reach 95%. The generating device determines that among the audio fingerprints of BGM1, BGM2, and BGM3, there is no first audio fingerprint whose similarity with the audio fingerprint to be identified reaches the preset similarity threshold, and obtains a first matching result, which indicates that BGM-A fails to match the first audio library.

[0122] S2012. The generating device determines whether the match is successful based on the first matching result. If so, execute S2011 for the next audio to be identified; otherwise, execute S2013.

[0123] S2013: The generating device matches the audio to be recognized with a preset second audio library to obtain a target matching result.

[0124] In the embodiment of the present application, the second audio library is established based on each second audio that fails to match the first audio library.

[0125] Since the matching process of the audio to be identified and the preset second audio library is the same as the matching process of the audio to be identified and the preset first audio library, it will not be repeated here.

[0126] S202: If the generating device determines, based on the target matching result, that the audio to be identified successfully matches at least one second audio in the second audio library, and there is a target second audio whose cumulative number of successful matches reaches a preset threshold value in the at least one second audio, then the target second audio is obtained.

[0127] For example, assuming that the preset threshold value is 10, the second audio library contains BGM4, the generating device determines based on the target matching result that BGM-A matches BGM4 in the second audio library successfully, and the cumulative number of successful matches of BGM4 reaches 10, then BGM4 is used as the target second audio.

[0128] For another example, assuming that the preset threshold value is 10, the second audio library contains BGM4 and BGM5, the generating device determines based on the target matching result that BGM-A matches BGM4 and BGM5 in the second audio library successfully, and the cumulative number of successful matches of BGM4 reaches 10, then BGM4 is used as the target second audio.

[0129] S203: The generating device transfers the target second audio as a new first audio to the first audio library.

[0130] For example, the generating device transfers BGM4 as a new first audio to the first audio library.

[0131] Below, S201-S203 are described with reference to a specific embodiment.

[0132] Take BGM-B as an example, see Figure 4 As shown, if BGM-B fails to match the first audio library, the generating device matches BGM-B with the second audio library. If BGM-B matches BGM-X in the second audio library successfully, and the cumulative number of successful matches of BGM-X reaches a preset threshold, BGM-X is transferred to the first audio library as a new first audio.

[0133] In some embodiments, if the generating device determines that the audio to be identified fails to match the second audio library based on the target matching result, then when it is determined that the audio to be identified meets the preset audio detection conditions, the audio to be identified is transferred to the second audio library as a new second audio.

[0134] Specifically, the following two methods may be used to determine whether the audio to be identified meets the preset audio detection condition, but are not limited to:

[0135] Method A:

[0136] The generating device determines that the audio to be recognized meets the preset audio detection condition, including at least one of the following operations:

[0137] Operation a1: If the audio to be identified includes an audio segment of a specified type and the duration of the audio segment reaches a preset duration threshold, the generating device determines that the audio to be identified meets the audio detection condition.

[0138] In the embodiment of the present application, the audio to be recognized may include but is not limited to one or more of the following types of audio clips: voice, singing, music, silence, noise, machine sound, ambient sound, etc.

[0139] For example, assuming that the specified type of audio clip is a music or singing type audio clip, the preset duration threshold is 3 minutes, and BGM-A contains a voice type audio clip 1 and a music type audio clip 2, wherein the duration of audio clip 2 is 4 minutes. At this time, the duration of audio clip 2 in BGM-A reaches the preset duration threshold of 3 minutes, and the generating device determines that BGM-A meets the audio detection conditions.

[0140] Operation a2: If the audio to be identified includes an audio segment of a specified type and the duration ratio of the audio segment reaches a preset ratio threshold, the generating device determines that the audio to be identified meets the audio detection condition.

[0141] The duration ratio is used to represent the ratio of the duration of the audio segment to the total duration of the audio to be recognized.

[0142] For example, assuming that the specified type of audio clip is a music or singing type audio clip, the preset ratio threshold is 80%, and BGM-A contains a voice type audio clip 1 and a music type audio clip 2, wherein the duration of audio clip 1 is 1 minute, and the duration of audio clip 2 is 4 seconds. At this time, the duration ratio of audio clip 2 in BGM-A is 80%, and the duration ratio of audio clip 2 in BGM-A reaches the preset ratio threshold of 80%, and the generating device determines that BGM-A meets the audio detection condition.

[0143] Operation a3: If at least one audio segment contained in the audio to be recognized is of a specified type, the generating device determines that the audio to be recognized meets the audio detection condition.

[0144] For example, assuming that the specified type of audio clip is a music or singing type audio clip, BGM-A contains a singing type audio clip 1 and a music type audio clip 2. At this time, the audio clip 1 and the audio clip 2 contained in BGM-A are both of the specified type, and the generating device determines that BGM-A meets the audio detection conditions.

[0145] For another example, assume that the specified type of audio clip is a music or singing type audio clip, and BGM-A contains a singing type audio clip 1. At this time, the audio clip 1 contained in BGM-A is of the specified type, and the generating device determines that BGM-A meets the audio detection conditions.

[0146] It should be noted that in the embodiment of the present application, a preset audio event detection algorithm can be used to determine whether the audio to be identified meets the preset audio detection conditions. The audio event detection algorithm can be used but is not limited to artificial neural network (ANN), hidden Markov model (HMM), etc.

[0147] In the above embodiment, when the audio to be recognized fails to match the second audio library, the audio to be recognized that meets the duration or duration ratio requirements, and the audio to be recognized without superimposed audio clips such as voice, environmental noise (for example, wind, rain, etc.), can be transferred to the second audio library as a new second audio.

[0148] Method B: To improve the matching efficiency, the machine learning technology can also be combined to determine whether the audio to be identified meets the preset audio detection condition. Specifically, the generating device determines that the audio to be identified meets the preset audio detection condition, including the following steps:

[0149] b1. The generating device inputs the audio to be recognized into the trained target audio detection model to obtain a detection prediction value.

[0150] The target audio detection model is obtained by training the audio detection model to be trained based on the audio annotation dataset. The audio annotation dataset represents a training dataset of annotated audio types.

[0151] It should be noted that in the implementation of this application, the target audio detection model can adopt but is not limited to a deep learning model. The detection prediction value can be expressed in levels or in numerical values, and this application does not limit this. In the following, only the use of numerical values ​​to express the detection prediction value is used as an example for explanation.

[0152] For example, the generating device inputs BGM-A into the trained target audio detection model and obtains a detection prediction value of BGM-A of 90.

[0153] b2. When the generating device determines that the detection prediction value reaches a preset prediction threshold value, it determines that the audio to be identified meets the audio detection condition.

[0154] For example, assuming that the preset prediction threshold is 85 and the detection prediction value of BGM-A is 90, the generating device determines that the detection prediction value of BGM-A reaches the preset prediction threshold and determines that BGM-A meets the audio detection condition.

[0155] The following describes a specific embodiment in which the matching between the audio to be recognized and the second audio library fails.

[0156] For example, the audio to be recognized is BGM-X. Figure 5 As shown, the first audio library does not contain the first audio, the generating device matches BGM-X with the first audio library, and if BGM-X fails to match the first audio library, then BGM-X is matched with the second audio library. The second audio library does not contain the second audio, and BGM-X fails to match the second audio library. When the generating device determines that BGM-X meets the preset audio detection conditions, BGM-X is added to the second audio library.

[0157] In some embodiments, in order to establish an audio library for audio content included in multimedia content, before executing S201, the generating device obtains the multimedia content to be identified, and extracts the audio to be identified from the multimedia content.

[0158] Specifically, the generating device obtains multimedia content to be identified, extracts audio content from the multimedia content to be identified, and when it is determined that the audio content meets a preset matching condition, uses the audio content as audio to be identified.

[0159] It should be noted that in the embodiment of the present application, the audio and video processing tool FFmpeg can be used but not limited to extract the audio file from the multimedia content to be identified. When the generating device determines that the audio file meets the preset matching condition, at least one of the operations a1 and a2 above is used, which will not be repeated here.

[0160] For example, see Figure 6As shown, the multimedia content to be identified includes images, text, and audio. The generating device obtains the multimedia content to be identified and extracts audio content from the multimedia content to be identified, wherein the audio content includes audio clips of voice type, audio clips of machine sound type, and audio clips of music type. The generating device determines that the audio content includes an audio clip of music type, and the duration of the audio clip of music type reaches a preset duration threshold value, and uses the audio content as the audio to be identified.

[0161] Furthermore, after the generating device transfers the second audio as the new first audio to the first audio library, the generating device also includes: the generating device uses the new first audio as the audio recognition result of the multimedia content, and records the multimedia identification of the multimedia content corresponding to the new first audio. The multimedia identification can be represented by, but not limited to, a serial number (Identity Document, ID).

[0162] Taking the audio to be identified as BGM-B as an example, assuming that BGM-B is extracted from video 1, the generating device transfers BGM-X as the new first audio to the first audio library, and then uses BGM-X as the audio recognition result of video 1.

[0163] In the above embodiment, a corresponding audio library can be established for the audio content included in the multimedia content, and when the multimedia content is subsequently acquired, the audio content can be identified, the multimedia content can be classified, and the multimedia content can be recommended based on the audio library.

[0164] It should be noted that the above embodiment only takes one multimedia content as an example, and the generating device may also obtain a multimedia content stream, and for each multimedia content in the multimedia content stream, use the above extraction method to obtain each audio to be recognized.

[0165] In some embodiments, during the use of audio, the user may partially modify the original audio, for example, accelerate or decelerate the original audio, insert other clips into the original audio, superimpose some other audio into the original audio, etc. Therefore, there may be a large number of similar first audios in the first audio. In order to achieve the association and aggregation of similar audios, in an embodiment of the present application, the generation device generates a corresponding first audio group based on every two first audios in the first audio library, and performs the following operations for each obtained first audio group:

[0166] The generating device determines two first audios contained in an audio group in each first audio group; obtains a multimedia identification set corresponding to each of the two first audios; if the number of repeated multimedia identifications in the two obtained multimedia identification sets reaches a preset number threshold, a new audio identification is generated corresponding to the two first audios.

[0167] Each multimedia identifier represents a multimedia content that successfully matches the corresponding first audio.

[0168] Take a first audio group (first audio group 1) in the first audio library as an example, assuming that the first audio group 1 includes BGM1 and BGM2, and the preset number threshold is 2. Fig. 7A As shown, the generating device determines that the first audio group 1 includes BGM1 and BGM2, and then the generating device obtains a multimedia identification set corresponding to BGM1, and obtains a multimedia identification set corresponding to BGM2, wherein the multimedia identification set corresponding to BGM1 includes multimedia identifications of video 1, video 2, video 3, video 4, video 8, and video 9, and the multimedia identification set corresponding to BGM2 includes multimedia identifications of video 3, video 4, video 6, and video 7. In the two obtained multimedia identification sets, the multimedia identifications that appear repeatedly are the multimedia identifications of video 3 and video 4. At this time, the number of the multimedia identifications that appear repeatedly reaches the preset number threshold value, see Figure 7B As shown, the generating device generates a new audio identifier BGM-1-2 corresponding to BGM1 and BGM2.

[0169] In some embodiments, see Figure 8 As shown, it is a flow chart of an audio recommendation method based on a generated first audio library provided in an embodiment of the present application, the method comprising the following steps:

[0170] S801. The generating device determines a recommended audio set from each first audio in response to an input operation triggered in the client based on the multimedia identification sets corresponding to each first audio contained in the first audio library.

[0171] The input operation triggered in the client includes, but is not limited to, the selection operation for the audio recommendation function. For example, see Fig.9A As shown, the generating device responds to a selection operation on the control 901 ("Template Library") triggered in the client.

[0172] Specifically, when S801 is executed, the following steps may be adopted but not limited to determine a recommended audio set from each first audio based on a multimedia identification set corresponding to each first audio contained in the first audio library:

[0173] S8011: The generating device obtains a set of multimedia identifiers corresponding to each first audio contained in the first audio library, and obtains an evaluation value corresponding to each first audio based on the obtained set of multimedia identifiers, wherein each evaluation value is used to characterize the usage status of the corresponding first audio.

[0174] Specifically, when executing S8011, the following steps may be adopted but are not limited to:

[0175] S80111: The generating device obtains the interactive state information of each corresponding multimedia content based on the obtained multimedia identification set corresponding to the first audio X. The first audio X is any first audio among the first audios.

[0176] In the embodiment of the present application, the interactive status information includes but is not limited to one or more of the following information: number of clicks, number of likes, number of comments, number of reposts, etc.

[0177] Taking the first audio X as BGM3 as an example, assuming that BGM3 corresponds to multimedia identifier set 1, and multimedia identifier set 1 includes multimedia identifiers of video 1, video 2, and video 3. The generating device obtains the number of likes of the corresponding video 1, video 2, and video 3 based on the obtained multimedia identifier set 1 corresponding to BGM3, wherein the number of likes of video 1 is 0, the number of likes of video 2 is 10, and the number of likes of video 3 is 100.

[0178] S80112. The generating device obtains the weight corresponding to each multimedia content based on the interaction status information of each multimedia content and the preset mapping relationship between the interaction status information and the weight.

[0179] Still taking the first audio X as BGM3 as an example, it is assumed that in the mapping relationship between the preset interactive state information and the weight, the weight of the multimedia content with a like count of 0 is 1, the weight of the multimedia content with a like count of 10 is 2, and the weight of the multimedia content with a like count of 100 is 3. The generating device obtains the weights corresponding to the video 1, video 2, and video 3 based on the like counts of the video 1, video 2, and video 3 and the mapping relationship, wherein the weights of the video 1, video 2, and video 3 are 0, 2, and 3, respectively.

[0180] S80113: The generating device obtains an evaluation value of the first audio X based on the weights corresponding to the respective multimedia contents.

[0181] Still taking the first audio X as BGM3 as an example, the generating device obtains an evaluation value of BGM3 of 0+2+3=5 based on the weights corresponding to video 1, video 2, and video 3.

[0182] It should be noted that, in the embodiment of the present application, in the preset mapping relationship between the interaction status information and the weight, different values ​​of the interaction status information may correspond to the same weight, that is, the weight of each multimedia content is the same.

[0183] Still taking the first audio X as BGM3 as an example, assuming that the mapping relationship between the preset interactive state information and the weight represents that the weights of various multimedia contents are the same, then the evaluation value of BGM3 obtained by the generating device is 1+1+1=3.

[0184] It should be noted that in the embodiment of the present application, the number of multimedia identification sets corresponding to the first audio X can also be used as the evaluation value of the first audio X. This application does not limit this and will not be repeated here.

[0185] S8022. The generating device sorts the first audios based on the evaluation values ​​corresponding to the first audios to obtain a target sequence, and sequentially selects a set number of first audios from the target sequence as a recommended audio set.

[0186] Taking the first audio as BGM1, BGM2, and BGM3 as an example, assuming that the corresponding evaluation values ​​of BGM1, BGM2, and BGM3 are 1, 2, and 3 respectively, and the set quantity is 2, the generating device sorts BGM1, BGM2, and BGM3 based on the corresponding evaluation values ​​of BGM1, BGM2, and BGM3 to obtain the target sequence: BGM3, BGM2, and BGM1, and selects BGM3 and BGM2 in turn from the target sequence as the recommended audio set.

[0187] S802: The generating device presents a recommended audio set in the client.

[0188] For example, see Fig. 9B As shown, the generating device presents a recommended audio set in the client, and the recommended audio set includes audios such as "Whale Transformed into a Lonely Island", "Lemon", and "Big Fish".

[0189] In some embodiments, in order to facilitate the subsequent classification, recommendation, and re-creation of multimedia content, the first audio library also includes audio knowledge information corresponding to each first audio, each audio knowledge information includes the audio identifier of the corresponding first audio, and each audio knowledge information may also include but is not limited to one or more of the audio name, audio tag, and related multimedia content tags of the corresponding first audio. For example, the audio knowledge information of BGM1 includes the audio identifier, audio name, audio tag, and related multimedia content tags of BGM1, where the audio name of BGM1 is "Big Fish", the audio tag of BGM1 is "Healing", and the related multimedia content tag of BGM1 is "Entertainment".

[0190] Each multimedia content has corresponding multimedia knowledge information, each multimedia knowledge information includes the multimedia identifier of the corresponding multimedia content, and each multimedia knowledge information may also include but is not limited to one or more of the multimedia name, keyword, multimedia tag, audio name, etc. of the corresponding multimedia content. For example, the multimedia knowledge information of video 1 is {What turned into blue buds?, #Spring Equinox Copywriting#, Strong Wind Blows}, where the video name of video 1 is "What turned into blue buds?", the keyword of video 1 is "Spring Equinox Copywriting", and the audio name of video 1 is "Strong Wind Blows".

[0191] In order to facilitate the subsequent classification and recommendation of multimedia content, if the audio to be identified successfully matches the first audio library, the generating device can add the audio knowledge information of the first audio that successfully matches the audio to be identified to the multimedia knowledge information of the multimedia content to which the audio to be identified belongs.

[0192] In order to further improve the first audio library, in an embodiment of the present application, according to a set statistical interval, for each first audio contained in the first audio library, the multimedia knowledge information of the multimedia content that successfully matches each first audio can be added to the audio knowledge information of the corresponding first audio.

[0193] In some embodiments, the generating device may periodically clean up the second audio library so as to timely discover new first audio. As an example, when the number of second audios contained in the second audio library exceeds a preset quantity threshold, the generating device may clean up the second audio library according to the storage time of each second audio. As another example, the generating device may clean up the second audio library according to the storage time of each second audio at a preset cleaning time interval.

[0194] Below, taking multimedia content as video as an example, the generation process of the popular sound effect library is explained.

[0195] See also Fig.10 As shown, the preset popular sound effect library does not contain any sound effect, and the preset temporary sound effect library does not contain any sound effect.

[0196] First, the generating device obtains the video 1 to be identified, and extracts the audio sound effect-C to be identified from the video 1. The generating device matches the sound effect-C with the popular sound effect library. If the matching fails, the sound effect-C is matched with the temporary sound effect library. If the matching fails again, if the sound effect-C meets the preset audio detection conditions, the sound effect-C is stored in the temporary sound effect library.

[0197] Then, the generating device obtains the video 2 to be identified, and extracts the audio sound effect-D to be identified from the video 1. The generating device matches the sound effect-D with the popular sound effect library. If the matching fails, the sound effect-D is matched with the temporary sound effect library. If the matching fails again, if the sound effect-D meets the preset audio detection conditions, the sound effect-D is stored in the temporary sound effect library.

[0198] Next, the generating device obtains the video 3 to be identified, and extracts the audio sound effect -C-1 to be identified from the video 3. The generating device matches the sound effect -C-1 with the popular sound effect library. If the match fails, the sound effect -C-1 is matched with the temporary sound effect library. The sound effect -C-1 is successfully matched with the sound effect C, and the cumulative number of successful matches of the sound effect C is updated.

[0199] Finally, the generating device obtains the video 4 to be identified, and extracts the audio sound effect-C-2 to be identified from the video 4. The generating device matches the sound effect-C-2 with the popular sound effect library. If the match fails, the sound effect-C-2 is matched with the temporary sound effect library. The sound effect-C-2 is successfully matched with the sound effect C, and the cumulative number of successful matches of the sound effect-C is updated.

[0200] When the cumulative number of successful matches of sound effect-C reaches a preset threshold, the generating device transfers sound effect-C to the popular sound effect library.

[0201] Based on the same inventive concept, the present application embodiment provides a device for generating an audio library. Fig.11 As shown, it is a schematic diagram of the structure of the audio library generation device 1100, which may include:

[0202] The matching unit 1101 is used to match the audio to be identified with a preset first audio library. If the match fails, the audio to be identified is matched with a preset second audio library to obtain a target matching result; wherein the second audio library is established based on each second audio that fails to match the first audio library;

[0203] The determining unit 1102 is configured to obtain the target second audio if it is determined based on the target matching result that the audio to be recognized successfully matches at least one second audio in the second audio library, and there is a target second audio whose cumulative number of successful matches reaches a preset threshold value in the at least one second audio;

[0204] The transfer unit 1103 is used to transfer the target second audio as a new first audio to the first audio library.

[0205] Optionally, the transfer unit 1103 is further used for:

[0206] If it is determined based on the target matching result that the audio to be identified fails to match the second audio library, then when it is determined that the audio to be identified meets the preset audio detection condition, the audio to be identified is transferred to the second audio library as a new second audio.

[0207] Optionally, when it is determined that the audio to be identified meets a preset audio detection condition, the determining unit 1102 is configured to perform at least one of the following operations:

[0208] If the audio to be identified contains an audio segment of a specified type and the duration of the audio segment reaches a preset duration threshold, it is determined that the audio to be identified meets the audio detection condition;

[0209] If the audio to be identified contains an audio clip of a specified type, and the duration ratio of the audio clip reaches a preset ratio threshold, it is determined that the audio to be identified meets the audio detection condition; wherein the duration ratio is used to represent the ratio of the duration of the audio clip to the total duration of the audio to be identified;

[0210] If at least one audio segment contained in the audio to be identified is of the specified type, it is determined that the audio to be identified meets the audio detection condition.

[0211] Optionally, when it is determined that the audio to be identified meets a preset audio detection condition, the determining unit 1102 is configured to:

[0212] Input the audio to be recognized into the trained target audio detection model to obtain a detection prediction value; wherein the target audio detection model is obtained by training the audio detection model to be trained based on the audio annotation dataset;

[0213] When it is determined that the detection prediction value reaches a preset prediction threshold value, it is determined that the audio to be identified meets the audio detection condition.

[0214] Optionally, before matching the audio to be recognized with a preset first audio library, the matching unit 1101 is further configured to:

[0215] Acquire multimedia content to be recognized, and extract audio to be recognized from the multimedia content;

[0216] After transferring the second audio as the new first audio to the first audio library, the transferring unit 1103 is further used for:

[0217] The new first audio is used as the audio recognition result of the multimedia content, and the multimedia identifier of the multimedia content is recorded corresponding to the new first audio.

[0218] Optionally, the device further includes an associating unit 1104, and the associating unit 1104 is used to:

[0219] Based on every two first audios in the first audio library, generate a corresponding first audio group;

[0220] For each first audio group obtained, perform the following operations respectively:

[0221] Determine two first audios included in one audio group among the first audio groups;

[0222] Obtaining a set of multimedia identifiers corresponding to the two first audios, wherein each multimedia identifier represents a multimedia content that successfully matches the corresponding first audio;

[0223] If the number of repeated multimedia identifiers in the two obtained multimedia identifier sets reaches a preset number threshold, a new audio identifier is generated corresponding to the two first audio associations.

[0224] Optionally, the device further includes a recommendation unit 1105, and the recommendation unit 1105 is used to:

[0225] In response to an input operation triggered in the client, based on the respective corresponding multimedia identifier sets of the respective first audios contained in the first audio library, a recommended audio set is determined from the respective first audios; wherein each multimedia identifier represents a multimedia content that successfully matches the corresponding first audio;

[0226] Renders a collection of recommended audios in the client.

[0227] Optionally, when determining the recommended audio set from each first audio based on the multimedia identification set corresponding to each first audio contained in the first audio library, the recommendation unit 1105 is specifically configured to:

[0228] Acquire a set of multimedia identifiers corresponding to each first audio contained in the first audio library, and obtain an evaluation value corresponding to each first audio based on the acquired set of multimedia identifiers; wherein each evaluation value is used to characterize the usage status of the corresponding first audio;

[0229] Based on the evaluation values ​​corresponding to the first audios, the first audios are sorted to obtain a target sequence, and a set number of first audios are selected in sequence from the target sequence as a recommended audio set.

[0230] Optionally, when obtaining evaluation values ​​corresponding to respective first audios based on the obtained multimedia identification sets, the recommendation unit 1105 is specifically configured to:

[0231] Based on the acquired multimedia identifier set corresponding to one of the first audios, acquiring the interaction status information of each corresponding multimedia content;

[0232] Based on the interactive state information of each multimedia content and the preset mapping relationship between the interactive state information and the weight, the weight corresponding to each multimedia content is obtained;

[0233] Based on the weights corresponding to the multimedia contents, an evaluation value of the first audio is obtained.

[0234] Optionally, when matching the audio to be recognized with a preset first audio library, the matching unit 1101 is specifically used to:

[0235] Based on a preset audio fingerprint extraction algorithm, extract the audio fingerprint of the audio to be identified, and obtain the audio fingerprint to be identified corresponding to the audio to be identified; wherein the audio fingerprint to be identified is used to characterize the audio features corresponding to the audio to be identified;

[0236] Based on the audio fingerprint extraction algorithm, extract the audio fingerprint of each first audio contained in the first audio library to obtain the first audio fingerprint corresponding to each first audio; wherein each first audio fingerprint is used to characterize the audio features of a corresponding first audio;

[0237] Calculate the similarity between the audio fingerprint to be identified and each of the first audio fingerprints obtained;

[0238] If there is at least one first audio fingerprint among the first audio fingerprints whose similarity with the audio fingerprint to be identified reaches a preset similarity threshold, it is determined that the audio to be identified successfully matches the first audio library.

[0239] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0240] Regarding the device in the above embodiment, the specific manner in which each unit executes the request has been described in detail in the embodiment of the method, and will not be elaborated here.

[0241] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0242] After introducing the method and device for generating an audio library according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0243] Fig.12 is a block diagram of an electronic device 1200 according to an exemplary embodiment, the device comprising:

[0244] Processor 1210;

[0245] A memory 1220 for storing instructions executable by the processor 1210;

[0246] The processor 1210 is configured to execute instructions to implement the method for generating an audio library in the embodiment of the present disclosure, for example Figure 2 , Figure 3A or Figure 8 Follow the steps shown in .

[0247] In an exemplary embodiment, a storage medium including operations is also provided, such as a memory 1220 including operations, and the operations can be executed by the processor 1210 of the electronic device 1200 to complete the above method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a portable compact disk read only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0248] Based on the same inventive concept, refer to Fig.13 As shown, the embodiment of the present application further provides a terminal device 1300, which can be an electronic device such as a smart phone, a tablet computer, a laptop or a PC.

[0249] The terminal device 1300 includes a display unit 1340, a processor 1380, and a memory 1320, wherein the display unit 1340 includes a display panel 1341, which is used to display information input by a user or information provided to a user and various operation interfaces of the terminal device 1300, and in the embodiment of the present application, is mainly used to display the operation interface, shortcut window, etc. of the application program installed in the terminal device 1300. Optionally, the display panel 1341 can be configured in the form of LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0250] The processor 1380 is used to read the computer program and then execute the method defined by the computer program. For example, the processor 1380 reads the application, thereby running the application on the terminal device 1300 and displaying the operation interface on the display unit 1340. The processor 1380 may include one or more general-purpose processors and may also include one or more DSPs (Digital Signal Processors) to perform related operations to implement the technical solutions provided in the embodiments of the present application.

[0251] The memory 1320 generally includes internal memory and external memory, and the internal memory can be RAM, ROM, and cache (CACHE), etc. The external memory can be a hard disk, an optical disk, a USB disk, a floppy disk or a tape drive, etc. The memory 1320 is used to store computer programs and other data, and the computer program includes an application program, etc. The other data may include data generated after the operating system or the application program is run, and the data includes system data (such as configuration parameters of the operating system) and user data. In the embodiment of the present application, program instructions are stored in the memory 1320, and the processor 1380 executes the program instructions in the memory 1320 to implement the generation method of the audio library discussed above.

[0252] In addition, the terminal device 1300 may also include a display unit 1340 for receiving input digital information, character information or contact touch operation / contactless gesture, and generating signal input related to user settings and function control of the terminal device 1300. Specifically, in the embodiment of the present application, the display unit 1340 may include a display panel 1341. The display panel 1341 is, for example, a touch screen, which can collect the user's touch operation on or near it (such as the user's operation on the display panel 1341 or on the display panel 1341 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the display panel 1341 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 1380, and can receive and execute the command sent by the processor 1380. In an embodiment of the present application, if a user selects a control in the operation interface, the touch detection device in the display panel 1341 detects the touch operation, and sends a signal corresponding to the detected touch operation to the touch controller. The touch controller converts the signal into touch point coordinates and sends them to the processor 1380. The processor 1380 determines the control selected by the user based on the received touch point coordinates.

[0253] The display panel 1341 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 1340, the terminal device 1300 may further include an input unit 1330, which may include but is not limited to one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick. Fig.13 In the figure, the input unit 1330 includes an image input device 1331 and other input devices 1332 as an example.

[0254] In addition to the above, the terminal device 1300 may also include a power supply 1390 for supplying power to other modules, an audio circuit 1360, a near field communication module 1370, and an RF circuit 1310. The terminal device 1310 may also include one or more sensors 1350, such as an accelerometer, a light sensor, a pressure sensor, etc. The audio circuit 1360 specifically includes a speaker 1361 and a microphone 1362, etc. For example, the user can use voice control, the terminal device 1300 can collect the user's voice through the microphone 1362, can be controlled by the user's voice, and when the user needs to be prompted, the corresponding prompt tone is played through the speaker 1361.

[0255] Based on the same inventive concept, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method for generating an audio library provided in various optional implementations of the above embodiments.

[0256] In some possible implementations, various aspects of the method for generating an audio library provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the method for generating an audio library according to various exemplary implementations of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 2 , Figure 3A or Figure 8 Follow the steps shown in .

[0257] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0258] The program product of the embodiment of the present application may adopt a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with a command execution system, device or device.

[0259] The readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with a command execution system, device, or device. Although the preferred embodiments of the present application have been described, once the basic creative concept is known to those skilled in the art, additional changes and modifications may be made to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0260] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating an audio library, characterized in that: The method includes: Acquire multimedia content to be identified, extract audio to be identified from the multimedia content, and match the audio to be identified with a preset first audio library. If the match fails, match the audio to be identified with a preset second audio library to obtain a target matching result; wherein the second audio library is established based on each second audio that fails to match the first audio library; If, based on the target matching result, it is determined that the audio to be recognized successfully matches at least one second audio in the second audio library, and among the at least one second audio, there is a target second audio whose cumulative number of successful matches reaches a preset threshold, then the target second audio is obtained; Transferring the target second audio as a new first audio to the first audio library; using the new first audio as the audio recognition result of the multimedia content, and recording the multimedia identifier of the multimedia content corresponding to the new first audio; Based on every two first audios in the first audio library, a corresponding first audio group is generated, and for each obtained first audio group, the following operations are performed respectively: Determine two first audios contained in one of the first audio groups, and obtain a set of multimedia identifiers corresponding to the two first audios, wherein each multimedia identifier represents a multimedia content that successfully matches the corresponding first audio; If the number of repeated multimedia identifiers in the two acquired multimedia identifier sets reaches a preset number threshold, a new audio identifier is generated corresponding to the two first audio associations.

2. The method according to claim 1, characterized in that Also includes: If it is determined based on the target matching result that the audio to be identified fails to match the second audio library, then when it is determined that the audio to be identified meets the preset audio detection condition, the audio to be identified is transferred to the second audio library as a new second audio.

3. The method according to claim 2, characterized in that The determining that the audio to be recognized meets a preset audio detection condition includes at least one of the following operations: If the audio to be identified includes an audio segment of a specified type and the duration of the audio segment reaches a preset duration threshold, it is determined that the audio to be identified meets the audio detection condition; If the audio to be identified contains an audio segment of a specified type, and the duration ratio of the audio segment reaches a preset ratio threshold, it is determined that the audio to be identified meets the audio detection condition; wherein the duration ratio is used to represent the ratio of the duration of the audio segment to the total duration of the audio to be identified; If at least one audio segment contained in the audio to be identified is of a specified type, it is determined that the audio to be identified meets the audio detection condition.

4. The method according to claim 2, characterized in that The step of determining whether the audio to be recognized meets a preset audio detection condition includes: Input the audio to be recognized into a trained target audio detection model to obtain a detection prediction value; wherein the target audio detection model is obtained by training the audio detection model to be trained based on an audio annotation dataset; When it is determined that the detection prediction value reaches a preset prediction threshold value, it is determined that the audio to be identified meets the audio detection condition.

5. The method according to any one of claims 1 to 4, characterized in that After recording the multimedia identifier of the multimedia content corresponding to the new first audio, the method further includes: In response to an input operation triggered in the client, based on the respective corresponding multimedia identification sets of the respective first audios contained in the first audio library, a recommended audio set is determined from the respective first audios; wherein each multimedia identification represents a multimedia content that successfully matches the corresponding first audio; The recommended audio set is presented in the client.

6. The method according to claim 5, characterized in that The determining a recommended audio set from each first audio based on the multimedia identification set corresponding to each first audio contained in the first audio library includes: Acquire a set of multimedia identifiers corresponding to each first audio contained in the first audio library, and obtain an evaluation value corresponding to each first audio based on the acquired set of multimedia identifiers; wherein each evaluation value is used to characterize the usage status of the corresponding first audio; Based on the evaluation values ​​corresponding to the respective first audios, the respective first audios are sorted to obtain a target sequence, and a set number of first audios are selected in sequence from the target sequence as a recommended audio set.

7. The method according to claim 6, characterized in that The obtaining, based on each acquired multimedia identification set, evaluation values ​​corresponding to each of the first audios includes: Based on the acquired multimedia identifier set corresponding to one of the first audios, acquiring the interaction status information of each corresponding multimedia content; Based on the interaction status information of each multimedia content and the preset mapping relationship between the interaction status information and the weight, obtaining the weight corresponding to each multimedia content; Based on the weights corresponding to the multimedia contents, an evaluation value of the first audio is obtained.

8. The method according to any one of claims 1 to 4, characterized in that The step of matching the audio to be recognized with a preset first audio library includes: Based on a preset audio fingerprint extraction algorithm, extract the audio fingerprint of the audio to be identified to obtain an audio fingerprint to be identified corresponding to the audio to be identified; wherein the audio fingerprint to be identified is used to characterize the audio features corresponding to the audio to be identified; Based on the audio fingerprint extraction algorithm, extract audio fingerprints from each first audio contained in the first audio library to obtain first audio fingerprints corresponding to each first audio; wherein each first audio fingerprint is used to characterize audio features of a corresponding first audio; Calculating the similarities between the audio fingerprint to be identified and each of the first audio fingerprints obtained; If there is at least one first audio fingerprint among the first audio fingerprints whose similarity with the audio fingerprint to be identified reaches a preset similarity threshold, it is determined that the audio to be identified is successfully matched with the first audio library.

9. A device for generating an audio library, characterized in that: include: A matching unit, used to obtain multimedia content to be identified, extract audio to be identified from the multimedia content, and match the audio to be identified with a preset first audio library. If the match fails, the audio to be identified is matched with a preset second audio library to obtain a target matching result; wherein the second audio library is established based on each second audio that fails to match the first audio library; a determination unit, configured to obtain the target second audio if it is determined based on the target matching result that the audio to be recognized matches the at least one second audio in the second audio library successfully, and there is a target second audio whose cumulative number of successful matches reaches a preset threshold value in the at least one second audio; The transfer unit is used to transfer the target second audio as a new first audio to the first audio library, and use the new first audio as the audio recognition result of the multimedia content, and record the multimedia identifier of the multimedia content corresponding to the new first audio; and based on every two first audios in the first audio library, generate a corresponding first audio group, and perform the following operations for each obtained first audio group: Determine two first audios contained in one of the first audio groups, and obtain a set of multimedia identifiers corresponding to the two first audios, wherein each multimedia identifier represents a multimedia content that successfully matches the corresponding first audio; If the number of repeated multimedia identifiers in the two acquired multimedia identifier sets reaches a preset number threshold, a new audio identifier is generated corresponding to the two first audio associations.

10. The device according to claim 9, characterized in that The transfer unit is also used for: If it is determined based on the target matching result that the audio to be identified fails to match the second audio library, then when it is determined that the audio to be identified meets the preset audio detection condition, the audio to be identified is transferred to the second audio library as a new second audio.

11. The device according to claim 10, characterized in that When it is determined that the audio to be recognized meets the preset audio detection condition, the determining unit is used to perform at least one of the following operations: If the audio to be identified includes an audio segment of a specified type and the duration of the audio segment reaches a preset duration threshold, it is determined that the audio to be identified meets the audio detection condition; If the audio to be identified contains an audio segment of a specified type, and the duration ratio of the audio segment reaches a preset ratio threshold, it is determined that the audio to be identified meets the audio detection condition; wherein the duration ratio is used to represent the ratio of the duration of the audio segment to the total duration of the audio to be identified; If at least one audio segment contained in the audio to be identified is of a specified type, it is determined that the audio to be identified meets the audio detection condition.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 8.

13. A computer-readable storage medium, characterized in that: It includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 8.

Images