Music recognition method, and training method and device of music feature extraction model

Abstract

The present disclosure provides a music recognition method and a training method and device of a music feature extraction model, and relates to the technical field of artificial intelligence, in particular to the technical field of audio processing and intelligent recommendation. The implementation scheme is as follows: determining a music segment in a to-be-recognized audio, the music segment being an audio segment containing a music component; extracting a first feature vector of the music segment; and determining at least one target music matching the to-be-recognized audio from a music library based on the first feature vector.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of audio processing and intelligent recommendation, and more particularly to a music recognition method and device, a training method and device of a music feature extraction model, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of humans, and includes both hardware technologies and software technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc.

[0003] Generally, a text-based search method is used to search for music, that is, by matching the search words (such as music name, singer, player, lyrics, etc.) input by the user with the attribute information of each music in the music library to obtain a search result. In the case where the user is not familiar with the music to be searched (for example, the user does not know the name, singer, etc. of the music to be searched), the above search method is difficult to obtain a satisfactory search result.

[0004] The methods described in this section can not have been previously conceived or made. Unless otherwise indicated herein, the mere fact that any method described in this section is set forth in this section should not be construed as being an admission that such method was ever previously conceived or made. Similarly, unless otherwise indicated herein, the mere fact that any problem is described in this section should not be construed as being an admission that such problem was ever previously considered in any prior art. SUMMARY

[0005] The present disclosure provides a music recognition method and device, a training method and device of a music feature extraction model, an electronic device, a computer readable storage medium, and a computer program product.

[0006] According to an aspect of the present disclosure, a music recognition method is provided, including: determining a music segment in a to-be-recognized audio, the music segment being an audio segment containing a music component; extracting a first feature vector of the music segment; and determining at least one target music matching the to-be-recognized audio from a music library based on the first feature vector.

[0007] According to one aspect of the present disclosure, a training method for a music feature extraction model is provided, which obtains a sample segment tuple, wherein the sample segment tuple includes a first sample music segment and a second sample music segment, and the sample segment tuple is annotated with a matching label, and the matching label is used to indicate whether the first sample music segment and the second sample music segment belong to the same sample music; extracts a first spectral feature of the first sample music segment and a second spectral feature of the second sample music segment; inputs the first spectral feature and the second spectral feature into the music feature extraction model respectively to obtain a first sample feature vector corresponding to the first sample music segment and a second sample feature vector corresponding to the second sample music segment; calculates a loss value of the music feature extraction model based on the matching label and the distance between the first sample feature vector and the second sample feature vector; and adjusts parameters of the music feature extraction model based on the loss value.

[0008] According to one aspect of the present disclosure, a music recognition device is provided, including: a first determination module, configured to determine a music clip in audio to be recognized, where the music clip is an audio clip containing music components; an extraction module, configured to extract a first feature vector of the music clip; and a second determination module, configured to determine at least one target music that matches the audio to be recognized from a music library based on the first feature vector.

[0009] According to one aspect of the present disclosure, a training device for a music feature extraction model is provided, comprising an acquisition module configured to acquire a sample segment tuple, wherein the sample segment tuple includes a first sample music segment and a second sample music segment, and the sample segment tuple is annotated with a matching label, wherein the matching label is used to indicate whether the first sample music segment and the second sample music segment belong to the same sample music; an extraction module configured to extract a first spectral feature of the first sample music segment and a second spectral feature of the second sample music segment; an input module configured to input the first spectral feature and the second spectral feature into the music feature extraction model, respectively, to obtain a first sample feature vector corresponding to the first sample music segment and a second sample feature vector corresponding to the second sample music segment; a calculation module configured to calculate a loss value of the music feature extraction model based on the matching label and a distance between the first sample feature vector and the second sample feature vector; and an adjustment module configured to adjust parameters of the music feature extraction model based on the loss value.

[0010] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute any of the methods of the above aspects.

[0011] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute any of the above methods.

[0012] According to one aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method of any one of the above aspects when executed by a processor.

[0013] According to one or more embodiments of the present disclosure, music recognition based on music clips in the audio to be recognized can avoid recognition errors and unnecessary computation caused by recognizing non-music clips, thereby improving the accuracy and computational efficiency of music recognition.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0016] Figure 1 A schematic diagram illustrating an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0017] Figure 2 A flowchart of a music recognition method according to an embodiment of the present disclosure is shown;

[0018] Figure 3A A schematic diagram showing a music clip in an audio data stream according to an embodiment of the present disclosure;

[0019] Figure 3B A schematic diagram showing a music clip in an audio file according to an embodiment of the present disclosure is shown;

[0020] Figure 4 A flowchart of a method for training a music feature extraction model according to an embodiment of the present disclosure is shown;

[0021] Figure 5 A schematic diagram of a music feature extraction model according to some embodiments of the present disclosure is shown;

[0022] Figure 6 Schematic diagram showing a music feature extraction model according to other embodiments of the present disclosure;

[0023] Figure 7 A schematic diagram illustrating a music recognition process according to an embodiment of the present disclosure is shown;

[0024] Figure 8 shows a structural block diagram of a music recognition device according to an embodiment of the present disclosure;

[0025] Figure 9 A structural block diagram of a training device for a music feature extraction model according to an embodiment of the present disclosure is shown; and

[0026] Figure 10 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0029] The terms used in the descriptions of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0030] In this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0031] In a case where a user is not familiar with the music to be searched (for example, the user does not know the name, singer, and the like of the music to be searched), a text-based music search method is difficult to obtain a satisfactory search result for the user. Therefore, it is desirable to provide a method of searching for music, that is, a user inputs an audio, and determines target music corresponding to the audio by recognizing the audio.

[0032] In the related art, generally, overall features of the audio input by the user are extracted, the target music corresponding to the audio is determined by matching the features with features of each music stored in a music library, and then information (for example, information such as the name of the music, the singer / performer, the lyricist / composer, the lyrics, and the like) of the target music is returned to the user. However, in the above-described scheme, the audio input by the user can not be complete music. For example, the first few seconds of the audio input by the user can be only environmental noise, and can not include a music component. In this case, directly recognizing the overall audio input by the user will result in inaccurate recognition results, and increase the amount of calculation.

[0033] To this end, embodiments of the disclosure provide a music recognition method and a training method of a music feature extraction model, which can achieve accurate and efficient music recognition.

[0034] Embodiments of the disclosure will be described in detail below with reference to the accompanying drawings.

[0035] Figure 1 A schematic diagram of an example system 100 in which the various methods and apparatus described herein can be implemented in accordance with embodiments of the disclosure is shown. Referring to Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 that couple the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0036] In embodiments of the disclosure, the server 120 can run one or more services or software applications that enable the execution of a music recognition method and / or a training method of a music feature extraction model.

[0037] In certain embodiments, the server 120 can also provide other services or software applications that can include non-virtual environments and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0038] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may, in turn, utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0039] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to perform music identification. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure can support any number of client devices.

[0040] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing a variety of different applications, such as various internet-related applications, communication applications (such as email applications), and short message service (SMS) applications, and may use various communication protocols.

[0041] The network 110 may be any type of network known to those skilled in the art that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.

[0042] Server 120 may include one or more general-purpose computers, specialized server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0043] The computing units in the server 120 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

[0044] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0045] In some embodiments, server 120 may be a distributed system server or a server integrated with blockchain. Server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host equipped with artificial intelligence technology. A cloud server is a host product within the cloud computing service system that addresses the management difficulties and poor scalability of traditional physical hosts and virtual private servers (VPS) services.

[0046] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as music files. The databases 130 may reside in a variety of locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0047] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0048] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in accordance with the present disclosure.

[0049] For the purpose of the embodiments of this disclosure, Figure 1 In the example of , client devices 101, 102, 103, 104, 105 and 106 may include a client application for music recognition. The client application may be, for example, an application that needs to be downloaded and installed before running, or a music website that can be accessed through a browser, or a lightweight applet that runs in a host application, etc. The client application may provide various functions based on music, such as music playback, music upload / download, text-based music search, music recognition function provided by the embodiment of the present disclosure, etc. Correspondingly, server 120 may be a server used together with the client application. Server 120 may provide music services to the client applications running in client devices 101, 102, 103, 104, 105 and 106 based on the music files stored in database 130 (i.e., "music library").

[0050] In an embodiment of the present disclosure, the server 120 may provide music recognition services to client applications running in the client devices 101 , 102 , 103 , 104 , 105 , and 106 .

[0051] According to some embodiments, a user can upload the audio to be identified to the server 120 through a client application running in the client devices 101, 102, 103, 104, 105 and 106 to initiate a music identification request. The server 120, in response to the music identification request, performs the music identification method of the embodiments of the present disclosure to determine at least one target music matching the audio to be identified from the stored music library, and returns the information of the target music (e.g. music name, singer / performer, lyricist / composer, lyrics, etc.) to the client device so that the client device presents the information of the target music to the user. Specifically, the target music can be a song sung by a singer, or pure music, etc.

[0052] According to some other embodiments, the server 120 can provide the music library itself or the address or access interface of the music library to the client application so that the client application can directly access the music library. In the case that the client can directly access the music library, the user can input the audio to be identified into the client application running in the client devices 101, 102, 103, 104, 105 and 106, and the client application performs (i.e. the client device performs) the music identification method of the embodiments of the present disclosure to determine at least one target music matching the audio to be identified from the music library, and presents the information of the target music to the user.

[0053] Figure 2 A flowchart of the music identification method 200 according to the embodiments of the present disclosure is shown. The method 200 can be performed at a server (e.g. the server 120 shown in Figure 1 ) or at a client device (e.g. the client devices 101, 102, 103, 104, 105 and 106 shown in Figure 1 ). That is, the performer of each step of the method 200 can be the server 120 shown in Figure 1 , or the client devices 101, 102, 103, 104, 105 and 106 shown in Figure 1 .

[0054] As shown in Figure 2 , the method 200 includes:

[0055] Step 210, determining a music segment in the audio to be identified, the music segment being an audio segment containing music component;

[0056] Step 220, extracting a first feature vector of the music segment; and

[0057] Step 230, determining at least one target music matching the audio to be identified from the music library based on the first feature vector.

[0058] According to the embodiments of the present disclosure, music recognition is performed based on music clips in the audio to be recognized, which can avoid recognition errors and unnecessary calculations caused by recognizing non-music clips, and improve the accuracy and calculation efficiency of music recognition.

[0059] The various steps of method 200 are described in detail below.

[0060] In step 210 , a music segment in the audio to be recognized is determined.

[0061] The audio to be recognized is typically input by the user. In some embodiments, an audio data stream recorded by the user in real time can be used as the audio to be recognized. For example, the user can record a piece of music that they accidentally heard and found interesting, or record themselves humming, and the audio data stream obtained from the recording is the audio to be recognized.

[0062] In other embodiments, a user-entered audio file can also be used as the audio to be recognized. Specifically, the user can directly enter the audio file itself or specify the address of the audio file. For example, a user may accidentally hear a piece of music that interests them, but the network environment at the time is poor, making it impossible to upload the recorded audio to the server for music recognition in real time. The user can record the music of interest and create an audio file. After the network environment is restored, the audio file can be uploaded for music recognition. The audio file will then serve as the audio to be recognized.

[0063] The audio to be identified may include audio signals generated by one or more sound sources (i.e., objects that make sounds). In the embodiments of the present disclosure, the audio signal generated by each sound source can be recorded as a component of the audio to be identified. Depending on whether the vibration of the sound source is regular or not, the components of the audio can be divided into two categories: music (or "musical sound") and noise. Specifically, music refers to the audio signal generated by the regular vibration of the sound source, and noise refers to the audio signal generated by the irregular vibration of the sound source.

[0064] In some cases, the audio to be identified may not be entirely music. For example, some segments of the audio to be identified input by the user may simply be ambient noise, without containing any musical elements. In an embodiment of the present disclosure, audio segments containing musical elements are recorded as "music segments", and audio segments not containing musical elements are recorded as "non-music segments". Non-music segments will affect the accuracy and efficiency of music recognition. Therefore, in step 210, the music segments in the audio to be identified are determined, and in subsequent steps 220 and 230, music recognition is performed based on the music segments determined in step 210, thereby avoiding recognition errors and unnecessary computation caused by identifying non-music segments.

[0065] According to some embodiments, when the audio to be identified is an audio data stream, the following steps can be followed to determine a music segment in the audio to be identified: sequentially intercepting audio frames of a preset duration from the audio data stream; for each intercepted audio frame, detecting whether the audio frame contains a music component; and in response to the number of consecutive audio frames containing music components reaching a first threshold, determining the audio data after the multiple audio frames as a music segment. This disclosure does not limit the value of the first threshold.

[0066] In the above embodiment, each audio frame in the audio data stream is sequentially detected to determine whether it contains a musical component. When N consecutive audio frames (N being a first threshold) are detected to contain musical components, the subsequent audio data is used as the music segment to be identified. Based on this embodiment, there is no need to cache the detected audio frames, thereby saving storage space. Furthermore, there is no need to perform frame segmentation and musical component detection on the subsequent audio data, thereby improving computational efficiency and real-time performance.

[0067] It should be noted that the present disclosure does not limit the value of the preset duration. According to some embodiments, in order to achieve real-time music recognition of the audio data stream, the preset duration can be set to a smaller value, such as 10ms, 20ms, etc.

[0068] According to some embodiments, a preset music detection model may be used to detect whether an audio frame contains music. Specifically, spectral features of the audio frame may be extracted, and then the spectral features may be input into the preset music detection model to obtain a detection result output by the music detection model indicating whether the audio frame contains music.

[0069] The spectral features of the audio frame may be, for example, a spectrogram, Mel Frequency Cepstral Coefficents (MFCCs), etc., but are not limited thereto.

[0070] The music detection model takes the spectral features of an audio frame as input and outputs a result indicating whether the audio frame contains music. Specifically, the music detection model outputs a probability p that the audio frame contains music. If the probability p is greater than or equal to a threshold P, the audio frame is considered to contain music.

[0071] The music detection model can be any neural network model, and this disclosure does not limit the structure of the music detection model. In some embodiments, the music detection model can be implemented as a convolutional neural network including multiple convolutional layers, where the stride of the convolution kernel of each convolutional layer can be, for example, 2, and the last convolutional layer of the model outputs the probability p that the audio frame contains a music component.

[0072] Figure 3AFIG. 1 shows a schematic diagram of determining a music segment in an audio data stream according to an embodiment of the present disclosure. Figure 3A , the audio to be identified is the audio data stream 300A, the duration of a single audio frame (ie, the “preset duration” above) is 10 ms, and the first threshold value of the number of audio frames is 8.

[0073] exist Figure 3A In the embodiment shown, first, a 10ms audio frame 301 is intercepted from the audio data stream 300A, the spectral features of the audio frame 301 are extracted, and its spectral features are input into a preset music detection model, and the music detection model outputs a probability p1 that the audio frame 301 contains a music component. Since p1 is less than the threshold value P, it is determined that the audio frame 301 does not contain a music component. Subsequently, the next audio frame 302 is intercepted from the audio data stream 300A, the spectral features of the audio frame 302 are extracted, and its spectral features are input into the music detection model, and the music detection model outputs a probability p2 that the audio frame 302 contains a music component. Since p2 is greater than the threshold value P, it is determined that the audio frame 302 contains a music component (in Figure 3A , audio frames containing music components are shaded. Similarly, audio frame 303 is determined to contain no music components, audio frames 304 and 305 contain music components, audio frames 306 and 307 do not contain music components, and audio frames 308-315 contain music components. After audio frame 315 is determined to contain music components, the number of consecutive audio frames containing music components (i.e., audio frames 308-315) reaches eight (i.e., reaching the first threshold), and therefore subsequent audio data 316 is determined to be a music segment for music recognition.

[0074] According to other embodiments, when the audio to be identified is an audio file, the following steps can be followed to determine the music segment in the audio to be identified: divide the audio file into multiple audio frames; for each of the multiple audio frames, detect whether the audio frame contains a music component; determine one or more consecutive audio frames containing music components as a candidate music segment to obtain at least one candidate music segment corresponding to the audio file; and determine the candidate music segment with the largest number of audio frames included in the at least one candidate music segment as the music segment.

[0075] In the above embodiment, determining the music segment by analyzing the entire audio file is helpful to improve the accuracy of music recognition.

[0076] It should be noted that the present disclosure does not impose any restrictions on the duration of each audio frame obtained by dividing the audio file. The duration of the audio frames obtained by dividing the audio file may be the same as or different from the duration of the audio frames obtained by intercepting the audio data stream in the above embodiment (i.e., the preset duration). According to some embodiments, the duration of the audio frames of the audio file can be set to a larger value (greater than the preset duration above), such as 1s, thereby reducing the number of audio frames included in the audio file and improving computing efficiency.

[0077] According to some embodiments, a preset music detection model may be used to detect whether an audio frame contains music. Specifically, spectral features of the audio frame may be extracted, and then the spectral features may be input into the preset music detection model to obtain a detection result output by the music detection model indicating whether the audio frame contains music.

[0078] The spectral features of the audio frame may be, for example, a spectrogram, Mel Frequency Cepstral Coefficents (MFCCs), etc., but are not limited thereto.

[0079] The music detection model takes the spectral features of an audio frame as input and outputs a result indicating whether the audio frame contains music. Specifically, the music detection model outputs a probability p that the audio frame contains music. If the probability p is greater than or equal to a threshold P, the audio frame is considered to contain music.

[0080] The music detection model can be any neural network model, and this disclosure does not limit the structure of the music detection model. In some embodiments, the music detection model can be implemented as a convolutional neural network including multiple convolutional layers, where the stride of the convolution kernel of each convolutional layer can be, for example, 2, and the last convolutional layer of the model outputs the probability p that the audio frame contains a music component.

[0081] Figure 3B FIG. 1 shows a schematic diagram of determining a music segment in an audio file according to an embodiment of the present disclosure. Figure 3B In the example, the audio to be recognized is the audio file 300B, and the duration of a single audio frame is 1 second.

[0082] exist Figure 3BIn the illustrated embodiment, audio file 300B is first divided into 27 audio frames, namely, audio frames 351-377. Audio frames 351-377 are respectively input into the music detection model, and it is determined that audio frames 352-355, 358-366, 370, and 375-376 contain musical components, while the other audio frames do not. Audio frames 352-355 are selected as candidate music segment 1, audio frames 358-366 are selected as candidate music segment 2, audio frame 370 is selected as candidate music segment 3, and audio frames 375-376 are selected as candidate music segment 4. Among the four candidate music segments, candidate music segment 2 includes the largest number of audio frames, and therefore is selected as the music segment for music recognition.

[0083] After determining the music segment in the audio to be recognized in step 210, step 220 may be performed to extract a first feature vector of the music segment.

[0084] According to some embodiments, spectral features of a music clip may be extracted; and the spectral features may be input into a preset music feature extraction model to obtain a first feature vector of the music clip output by the music feature extraction model.

[0085] According to the above embodiment, using a music feature extraction model to convert the spectral features of a music clip into a first eigenvector can reduce the feature dimension (the first eigenvector has a smaller dimension than the spectral features), thereby improving the information density and robustness of the features. Determining the target music that matches the audio to be recognized based on the first eigenvector can improve the accuracy of music recognition.

[0086] The spectral features of the music clip may be, for example, a spectrogram, a Mel-frequency cepstral coefficient, etc., but are not limited thereto.

[0087] The music feature extraction model can be any neural network model, and the present disclosure does not limit the structure of the music feature extraction model. According to some embodiments, the music feature extraction model can be trained by the following method 400.

[0088] According to some embodiments, the music clip can be further divided into multiple music sub-segments of the same duration (e.g., 1 second), and the spectral features of each music sub-segment are input into the music feature extraction model to obtain the first sub-feature vector corresponding to each music sub-segment. Accordingly, the first feature vector of the music clip is the set of the first sub-feature vectors. This allows the length of the vector corresponding to the first feature vector to be unified (to the same length as the second feature vector described below), thereby facilitating the calculation of the distance between the first feature vector and the second feature vector in the subsequent step 230.

[0089] According to another embodiment, a music sub-clip with a fixed time length (e.g. 1s) can also be cropped from the music clip, and the spectral features of the music sub-clip can be input into the music feature extraction model to obtain a first sub-feature vector corresponding to the music sub-clip. The first sub-feature vector is the first feature vector of the music clip. In this way, the first feature vector can be adjusted to a fixed length (same as the length of the second feature vector below), so as to facilitate the calculation of the distance between the first feature vector and the second feature vector in the subsequent step 230.

[0090] According to some embodiments, after the first feature vector of the music clip is extracted through step 220, step 230 can be performed to determine at least one target music matching the audio to be identified from the music library based on the first feature vector.

[0091] According to some embodiments, the music library includes a plurality of candidate musics, which can be any type of music, such as songs sung by a singer, pure music, etc., but are not limited thereto. Step 230 can further include steps 232 and 234 as follows.

[0092] In step 232, for each of the plurality of candidate musics, steps 232a-232c are performed as follows:

[0093] In step 232a, a plurality of second feature vectors corresponding to the candidate music are obtained, wherein the candidate music is divided into a plurality of clips, and the plurality of clips correspond to the plurality of second feature vectors respectively.

[0094] In step 232b, the distance between the first feature vector and each of the plurality of second feature vectors is calculated respectively.

[0095] In step 232c, based on the above distance, the matching degree between the audio to be identified and the candidate music is determined.

[0096] In step 232a, according to some embodiments, the second feature vector of a clip can be obtained by extracting the spectral features of the clip and then inputting the spectral features of the clip into a preset music feature extraction model, for example. It can be understood that the music feature extraction model used to extract the second feature vector is the same model as the aforementioned music feature extraction model used to extract the first feature vector.

[0097] In step 232b, according to some embodiments, the distance between the first feature vector and the second feature vector can be the Euclidean distance. In other embodiments, the distance between the first feature vector and the second feature vector can also be the cosine distance, Manhattan distance, etc.

[0098] In step 232c, according to some embodiments, segments corresponding to the second feature vector whose distance from the first feature vector is less than a third threshold may be considered matching segments, and the ratio of the number of matching segments in the candidate music to the total number of segments included in the candidate music may be determined as the degree of match between the candidate music and the audio to be identified. The present disclosure does not limit the value of the third threshold.

[0099] For example, the first feature vector of the music segment in the audio to be identified is a, and the candidate music music1 includes 10 segments, namely seg1-seg 10 , the third threshold is thr3. seg1-seg 10 The corresponding second eigenvectors are b1-b 10 . Calculate the first eigenvector a and the second eigenvector b1-b respectively 10 The Euclidean distance between a and b1-b is obtained. 10 The Euclidean distances are d1-d 10 , where d1, d4, and d9 are less than the third threshold thr3, the corresponding segments seg1, seg4, and seg9 are matching segments, and the number of matching segments is 3. The matching degree between the candidate music music1 and the music to be identified is the ratio of the number of matching segments to the total number of segments included in the candidate music, that is, 3 / 10 = 0.3.

[0100] Based on the above embodiment, by determining the ratio of the number of matching segments in the candidate music to the total number of multiple segments included in the candidate music as the matching degree of the candidate music and the audio to be identified, the matching of the audio to be identified and the candidate music can be considered as a whole, avoiding errors caused by considering only the matching of a single segment.

[0101] After obtaining the matching degree between the audio to be recognized and each candidate music through step 232 (including steps 232a-232c), step 234 may be executed.

[0102] In step 234, one or more candidate music pieces with the greatest matching degree or a matching degree greater than a second threshold are selected as at least one target music piece. Furthermore, information (e.g., music title, singer / performer, lyricist / composer, lyrics, etc.) of the at least one target music piece may be returned to the user.

[0103] It should be noted that in step 234, the target music is determined based on the degree of match. This disclosure does not limit the specific strategy for determining the target music based on the degree of match. According to some embodiments, the target music can be determined based on the ranking of the degree of match. For example, the candidate music with the highest degree of match can be used as the target music, the three candidate music with the highest degree of match can be used as the target music, and so on.

[0104] According to another embodiment, the target music can also be determined according to the numerical value of the matching degree. For example, the candidate music with a matching degree greater than 0.6 (second threshold) is taken as the target music, the candidate music with a matching degree greater than 0.7 (second threshold) is taken as the target music, and so on.

[0105] It can be understood that according to the set strategy, there can be a case where the target music cannot be obtained. For example, it is set that the candidate music with a matching degree greater than 0.8 is taken as the target music, but the matching degrees of all candidate music with the audio to be identified are less than 0.8, and thus there is no target music. In this case, a prompt message indicating that the music recognition fails can be returned to the user, such as “Sorry, no matching music found”, “Matching failed, please try again”, and the like.

[0106] In addition, it needs to be noted that the value of the second threshold and the number of target music are not limited by the present disclosure. Generally, the number of target music can be 1.

[0107] For example, after step 232, the matching degrees of the audio to be identified with the candidate music music1 – music5 are obtained as 0.3, 0.7, 0.65, 0.66, and 0.2 respectively. The candidate music with the largest matching degree, i.e., music2, can be taken as the target music. Alternatively, the two candidate music with the largest matching degrees, i.e., music2 and music4, can be taken as the target music. Alternatively, the candidate music with a matching degree greater than 0.6, i.e., music2, music3, and music4, can be taken as the target music. Alternatively, the candidate music with a matching degree greater than 0.8 can be taken as the target music, but in this case, there will be no target music, and accordingly, a prompt message “Sorry, no matching music found” can be returned to the user.

[0108] According to some embodiments, the method 200 can further include: obtaining new music in the network at a preset frequency; and adding the new music to the music library. In this way, new music in the network can be detected at a regular time, and the detected new music can be added to the music library, thereby improving the recognition effect of new music.

[0109] The new music in the network can be obtained by a script, for example.

[0110] According to the embodiments of the present disclosure, a training method of a music feature extraction model is also provided, based on which a music feature extraction model for accurately extracting music features can be obtained. Based on the music features extracted by the model, accurate and efficient music recognition can be achieved.

[0111] Figure 4A flowchart of a method 400 for training a music feature extraction model according to an embodiment of the present disclosure is shown. Method 400 can be executed, for example, on a server or on a client device. That is, each step of method 400 can be executed by either a server or a client device.

[0112] like Figure 4 As shown, the method 400 includes:

[0113] Step 410: Obtain a sample segment tuple, wherein the sample segment tuple includes a first sample music segment and a second sample music segment, and the sample segment tuple is annotated with a matching tag, which is used to indicate whether the first sample music segment and the second sample music segment belong to the same sample music;

[0114] Step 420: extracting a first spectral feature of the first sample music clip and a second spectral feature of the second sample music clip;

[0115] Step 430: Input the first spectral feature and the second spectral feature into a music feature extraction model to obtain a first sample feature vector corresponding to the first sample music clip and a second sample feature vector corresponding to the second sample music clip;

[0116] Step 440: Calculate the loss value of the music feature extraction model based on the matching label and the distance between the first sample feature vector and the second sample feature vector; and

[0117] Step 450: Adjust the parameters of the music feature extraction model based on the loss value.

[0118] According to an embodiment of the present disclosure, the music feature extraction model can map the spectral features of music to a new space to form a feature representation of the music in the new space. During the training process of the music feature extraction model, the loss value is calculated based on the matching label and the distance between the first sample feature vector and the second sample feature vector, so that the features output by the trained model can well judge whether two pieces of music match. Furthermore, music recognition based on the features output by the model can improve the accuracy of music recognition. Moreover, compared with the solution of directly using a neural network to output whether two pieces of music match (that is, whether they belong to the same music), the amount of calculation is smaller and the efficiency is higher.

[0119] The music feature extraction model can be any neural network model, and the present disclosure does not limit the structure of the music feature extraction model. For example, the music feature extraction model can be implemented as a convolutional neural network model.

[0120] The first sample music segment and the second sample music segment can be a segment of any music in the music library, and the time length of the first sample music segment and the second sample music segment is the same (for example, both are 1s). For example, 100 musics in the music library can be taken as sample musics, each sample music is divided into multiple segments with a time length of 1s, and then two segments are randomly selected from all the segments divided from the 100 sample musics, and the two segments are respectively taken as the first sample music segment and the second sample music segment.

[0121] According to some embodiments, the first spectral feature and the second spectral feature can be a spectrogram, a mel-frequency cepstral coefficient, etc., but are not limited thereto.

[0122] According to some embodiments, the sample segment tuple can include and only include the first sample music segment and the second sample music segment. That is, the sample segment tuple is a binary tuple. In this case, in step 440, the loss value of the music feature extraction model can be calculated according to the following formula (1), that is, the loss function of the music feature extraction model is the following formula (1):

[0123]

[0124] Wherein, N is the number of sample segment tuples. y is a matching label, if the first sample music segment and the second sample music segment belong to the same sample music, then y = 1; if the first sample music segment and the second sample music segment belong to different sample musics, then y = 0. d is the distance (for example, the Euclidean distance) between the first sample feature vector and the second sample feature vector. margin is a preset normal number.

[0125] Figure 5 The calculation process of the loss value of the music feature extraction model 500 is shown in the case where the sample segment tuple includes and only includes the first sample music segment and the second sample music segment. As shown in Figure 5 The first spectral feature ff1 of the first sample music segment is input into the music feature extraction model 500 to obtain the first sample feature vector sf1 corresponding to the first sample music segment. The second spectral feature ff2 of the second sample music segment is input into the music feature extraction model 500 to obtain the second sample feature vector sf2 corresponding to the second sample music segment. The loss value of the music feature extraction model 500 is calculated based on the distance d between the first sample feature vector sf1 and the second sample feature vector sf2 (for example, the loss value is calculated according to the above formula (1)).

[0126] According to some embodiments, the sample segment tuple may further include a third sample music segment in addition to the first sample music segment and the second sample music segment. The third sample music segment may be a different sample music segment from the first sample music segment if the first sample music segment and the second sample music segment are the same sample music, or may be the same sample music segment as the first sample music segment if the first sample music segment and the second sample music segment are different sample music segments.

[0127] Accordingly, the method 400 further includes: extracting a third spectrum feature of the third sample music segment; and inputting the third spectrum feature into a music feature extraction model to obtain a third sample feature vector corresponding to the third sample music segment.

[0128] In the case where the sample segment tuple is a triple, in step 440, the loss value of the music feature extraction model can be calculated based on the matching label, the distance between the first sample feature vector and the second sample feature vector, and the distance between the first sample feature vector and the third sample feature vector. For example, the loss value of the music feature extraction model can be calculated according to the following formula (2), that is, the loss function of the music feature extraction model is the following formula (2):

[0129] L=max(d(a,p)-d(a,n)+margin,0) (2)

[0130] Among them, a is the first sample feature vector. p is the sample feature vector corresponding to the sample music segment in the second sample music segment and the third sample music segment, which belongs to the same sample music as the first sample music segment. n is the sample feature vector corresponding to the sample music segment in the second sample music segment and the third sample music segment, which belongs to a different sample music than the first sample music segment. For example, the second sample music segment and the first sample music segment belong to the same sample music, and the third sample music segment and the first sample music segment belong to different sample music, then p is the second sample feature vector, and n is the third sample feature vector. d(a,p) represents the distance between a and p, and d(a,n) represents the distance between a and n. margin is a preset positive constant.

[0131] Figure 6 FIG. 6 shows a calculation process of the loss value of the music feature extraction model 600 when the sample segment tuple includes a first sample music segment, a second sample music segment, and a third sample music segment. Figure 6As shown, the first spectral feature ff1 of the first sample music clip is input into the music feature extraction model 600 to obtain the first sample feature vector sf1 corresponding to the first sample music clip. The second spectral feature ff2 of the second sample music clip is input into the music feature extraction model 600 to obtain the second sample feature vector sf2 corresponding to the second sample music clip. The third spectral feature ff3 of the third sample music clip is input into the music feature extraction model 600 to obtain the third sample feature vector sf3 corresponding to the third sample music clip. The loss value of the music feature extraction model 600 is calculated based on the distance d1 between the first sample feature vector sf1 and the second sample feature vector sf2, and the distance d2 between the first sample feature vector sf1 and the third sample feature vector sf3 (for example, the loss value is calculated according to the above formula (2)).

[0132] In step 450, the parameters of the music feature extraction model may be adjusted, for example, by a back-propagation algorithm.

[0133] The above steps 410-450 can be executed repeatedly until the loss value of the music feature extraction model is less than a certain threshold, and the training of the music feature extraction model is completed.

[0134] According to some embodiments, before step 420, method 400 may further include: performing data enhancement processing on the first sample music clip and the second sample music clip respectively to improve the diversity of the sample data, thereby improving the generalization ability and accuracy of the model.

[0135] According to some embodiments, when the sample segment tuple further includes a third sample music segment, method 400 may further include: performing data enhancement processing on the third sample music segment.

[0136] According to some embodiments, data augmentation processing may include at least one of the following: adding noise, adding room reverberation effects, performing time shifting, and adjusting volume. This can improve the model's ability to extract features for music containing noise, room reverberation, time shifting, and varying volume levels.

[0137] According to some embodiments, noise can be added by adjusting noise data (e.g., white noise, road noise, cafeteria noise, etc.) to the same volume as the first sample music clip (or the second sample music clip, or the third sample music clip), and then superimposing the noise data with the first sample music clip (or the second sample music clip, or the third sample music clip). In other words, in this embodiment, the superimposed noise data is additive noise.

[0138] According to some embodiments, an indoor reverberation effect can be added to the first sample music clip (or the second sample music clip, the third sample music clip) by convolving the first sample music clip (or the second sample music clip, the third sample music clip) with a preset reverberation excitation function.

[0139] According to some embodiments, time shifting may include shifting the first sample music segment (or the second sample music segment, or the third sample music segment) left or right within the sample music segment to which it belongs. For example, if the first sample music segment is the segment between 2 and 3 seconds in sample music 1, it may be shifted 100 ms to the left, resulting in an enhanced first sample music segment between 1.9 and 2.9 seconds in sample music 1.

[0140] According to some embodiments, adjusting the volume may include adjusting the first sample music segment and the second sample music segment (and the third sample music segment) to the same volume, that is, achieving volume normalization.

[0141] According to some embodiments, method 400 further includes: acquiring new music from the network at a preset frequency; and using the new music as sample music for training the music feature extraction model. This allows for expansion of the sample music. Further fine-tuning the music feature extraction model based on the expanded sample music can improve the accuracy and generalization of the music feature extraction model.

[0142] Figure 7 FIG. 1 shows a schematic diagram of a music recognition process according to an embodiment of the present disclosure. Figure 7 As shown, the music capture device 710 regularly captures new music from the network and stores the new music information (such as the identifier, name, performer / singer, lyricist / composer, lyrics, album name, era, style, etc.) in the music library 720. Furthermore, the new music is divided into multiple segments (for example, multiple segments of 1 second duration), and the spectral features of each segment are extracted. The spectral features are input into the music feature extraction model 730 to obtain a second feature vector for each segment. The second feature vector is stored in association with the identifier of the corresponding new music.

[0143] For example, if a newly added music piece has a duration of 60 seconds, it is divided into 60 segments of 1 second each. The spectral features of each segment are extracted and input into the music feature extraction model 730 to obtain 60 second feature vectors. These 60 second feature vectors are associated with the identifier of the newly added music piece and stored.

[0144] Furthermore, the music feature extraction model 730 can be fine-tuned based on the newly added music to improve the model's feature extraction performance. For example, the newly added music can be used as sample music, and each segment of the newly added music can be used as a sample music segment. These sample music segments can then be fed into the data enhancement device 720 for data enhancement processing. Data enhancement processing can include, for example, adding additive noise, adding room reverberation effects, performing time shifting, adjusting volume, etc. The music feature extraction model 730 can then be fine-tuned based on the enhanced sample music segments.

[0145] When a user inputs audio to be recognized, a music segment within the audio segment is identified and its spectral features are extracted. These spectral features are then input into a fine-tuned music feature extraction model 730 to obtain a first feature vector for the music segment. Subsequently, a music library query module 740 compares the first feature vector with the second feature vectors of each piece of music in the music library 720 to identify at least one target piece of music from the music library 720 as the music recognition result for the audio segment. The music recognition result is then returned to the user.

[0146] According to an embodiment of the present disclosure, a music recognition device is also provided. Figure 8 FIG. 8 shows a structural block diagram of a music recognition device 800 according to an embodiment of the present disclosure. Figure 8 As shown, the apparatus 800 includes:

[0147] The first determination module 810 is configured to determine a music segment in the audio to be identified, where the music segment is an audio segment containing music components;

[0148] An extraction module 820 is configured to extract a first feature vector of the music clip; and

[0149] The second determination module 830 is configured to determine at least one target music that matches the audio to be recognized from the music library based on the first feature vector.

[0150] According to the embodiments of the present disclosure, music recognition is performed based on music clips in the audio to be recognized, which can avoid recognition errors and unnecessary calculations caused by recognizing non-music clips, and improve the accuracy and calculation efficiency of music recognition.

[0151] According to an embodiment of the present disclosure, a training device for a music feature extraction model is also provided. Figure 9 FIG. 9 is a block diagram showing a structure of a training device 900 for a music feature extraction model according to an embodiment of the present disclosure. Figure 9 As shown, the apparatus 900 includes:

[0152] An acquisition module 910 is configured to acquire a sample segment tuple, wherein the sample segment tuple includes a first sample music segment and a second sample music segment, and the sample segment tuple is annotated with a matching tag, wherein the matching tag is used to indicate whether the first sample music segment and the second sample music segment belong to the same sample music;

[0153] An extraction module 920 is configured to extract a first spectral feature of the first sample music clip and a second spectral feature of the second sample music clip;

[0154] An input module 930 is configured to input the first spectral feature and the second spectral feature into the music feature extraction model respectively to obtain a first sample feature vector corresponding to the first sample music clip and a second sample feature vector corresponding to the second sample music clip;

[0155] a calculation module 940 configured to calculate a loss value of the music feature extraction model based on the matching label and the distance between the first sample feature vector and the second sample feature vector; and

[0156] The adjustment module 950 is configured to adjust the parameters of the music feature extraction model based on the loss value.

[0157] According to an embodiment of the present disclosure, the music feature extraction model can map the spectral features of music to a new space to form a feature representation of the music in the new space. During the training process of the music feature extraction model, the loss value is calculated based on the matching label and the distance between the first sample feature vector and the second sample feature vector, so that the features output by the trained model can well judge whether two pieces of music match. Furthermore, music recognition based on the features output by the model can improve the accuracy of music recognition. Moreover, compared with the solution of directly using a neural network to output whether two pieces of music match (that is, whether they belong to the same music), the amount of calculation is smaller and the efficiency is higher.

[0158] It should be understood that Figure 8 The modules or units of the apparatus 800 shown in FIG. 8 can be used in conjunction with the reference Figure 2 The steps in the method 200 described above correspond to the following. Figure 9 The modules of the apparatus 900 shown in FIG. 1 can be used in conjunction with the reference Figure 4 The steps in the method 400 described above correspond to each other. Thus, the operations, features, and advantages described above for method 200 are also applicable to apparatus 800 and its included modules and units, and the operations, features, and advantages described above for method 400 are also applicable to apparatus 900 and its included modules. For the sake of brevity, some operations, features, and advantages are not described here in detail.

[0159] While specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of each module discussed herein may be separated into multiple modules, and / or at least some functionality of multiple modules may be combined into a single module. For example, the calculation module 940 and the adjustment module 950 described above may be combined into a single module in some embodiments.

[0160] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 8 、 Figure 9 The various modules described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of the first determination module 810, the extraction module 820, the second determination module 820, the acquisition module 910, the extraction module 920, the input module 930, the calculation module 940, and the adjustment module 950 can be implemented together in a system on chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0161] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0162] refer to Figure 10 , a block diagram of an electronic device 1000 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0163] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0164] Multiple components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device that can input information to the device 1000. The input unit 1006 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 1007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 1008 can include but is not limited to a magnetic disk and an optical disk. The communication unit 1009 allows the device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TM devices, 802.11 devices, Wi-Fi devices, WiMAX devices, cellular communication devices, and / or the like.

[0165] The computing unit 1001 may be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as method 200 or method 400. For example, in some embodiments, method 200 or method 400 may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of method 200 or method 400 described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the method 200 or the method 400 in any other appropriate manner (eg, by means of firmware).

[0166] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0168] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0170] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0171] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0172] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0173] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. A music recognition method, comprising: Determining a music segment in the audio to be identified, wherein the music segment is an audio segment containing music components; extracting a first feature vector of the music clip; as well as Based on the first feature vector, determining at least one target music that matches the audio to be recognized from a music library; In response to the audio to be recognized being an audio data stream recorded in real time by a user, determining the music segment in the audio to be recognized includes: Sequentially intercepting audio frames of a preset duration from the audio data stream; For each audio frame extracted, detecting whether the audio frame contains a music component; and In response to the number of a plurality of consecutive audio frames containing music components reaching a first threshold, stopping framing and music component detection on subsequent audio data, and determining audio data subsequent to the plurality of audio frames that has not been detected for music components as the music segment, wherein the plurality of audio frames are not cached as the music segment; and In response to the audio to be identified being an audio file recorded and uploaded by a user, determining the music clip in the audio to be identified includes: Dividing the audio file into a plurality of audio frames according to a preset duration, wherein the preset duration for dividing the audio file is greater than a preset duration for intercepting audio frames from the audio data stream; For each of the plurality of audio frames of the audio file, detecting whether the audio frame contains a music component; determining one or more consecutive audio frames containing music components in the audio file as a candidate music segment to obtain at least one candidate music segment corresponding to the audio file, wherein each audio frame contained in each candidate music segment contains music components; and The candidate music segment that includes the largest number of audio frames among the at least one candidate music segment is determined as the music segment.

2. The method according to claim 1, wherein The detecting whether the audio frame contains music components comprises: extracting spectral features of the audio frame; and The spectral features are input into a preset music detection model to obtain a detection result output by the music detection model as to whether the audio frame contains music components.

3. The method according to claim 1, wherein Extracting the first feature vector of the music clip includes: Extracting spectral features of the music clip; and The spectral feature is input into a preset music feature extraction model to obtain a first feature vector of the music segment output by the music feature extraction model.

4. The method according to claim 1, wherein The music library includes a plurality of candidate music, and wherein, based on the first feature vector, determining at least one target music that matches the audio to be recognized from the music library includes: For each candidate music in the plurality of candidate music: Obtaining a plurality of second feature vectors corresponding to the candidate music, wherein the candidate music is divided into a plurality of segments, and the plurality of segments correspond to the plurality of second feature vectors respectively; respectively calculating a distance between the first eigenvector and each of the plurality of second eigenvectors; and Determining a degree of matching between the audio to be recognized and the candidate music based on the distance; as well as One or more candidate music with the maximum matching degree or a matching degree greater than a second threshold is used as the at least one target music.

5. The method according to claim 4, wherein Determining a degree of matching between the audio to be recognized and the candidate music based on the distance includes: taking a segment corresponding to a second feature vector whose distance from the first feature vector is less than a third threshold as a matching segment; and The ratio of the number of matching segments in the candidate music to the total number of the plurality of segments is determined as the matching degree.

6. The method according to any one of claims 1 to 5, further comprising: Get new music from the network according to the preset frequency; as well as Add the newly added music to the music library.

7. The method according to claim 3, wherein: The preset music feature extraction model is trained in the following way: Obtaining a sample segment tuple, wherein the sample segment tuple includes a first sample music segment and a second sample music segment, and the sample segment tuple is annotated with a matching tag, wherein the matching tag is used to indicate whether the first sample music segment and the second sample music segment belong to the same sample music; extracting a first spectral feature of the first sample music clip and a second spectral feature of the second sample music clip; Inputting the first spectral feature and the second spectral feature into an initial music feature extraction model respectively to obtain a first sample feature vector corresponding to the first sample music clip and a second sample feature vector corresponding to the second sample music clip; Calculating a loss value of the initial music feature extraction model based on the matching label and the distance between the first sample feature vector and the second sample feature vector; Based on the loss value, the parameters of the initial music feature extraction model are adjusted to obtain the preset music feature extraction model.

8. The method according to claim 7, before extracting the first spectral feature of the first sample music segment and the second spectral feature of the second sample music segment, further comprising: Data enhancement processing is performed on the first sample music clip and the second sample music clip respectively.

9. The method according to claim 7, wherein: The sample segment tuple further includes a third sample music segment, wherein the third sample music segment is a different sample music segment from the first sample music segment when the first sample music segment and the second sample music segment are the same sample music, and is the same sample music segment as the first sample music segment when the first sample music segment and the second sample music segment are different sample music; The method further comprises: extracting a third frequency spectrum feature of the third sample music segment; and Inputting the third spectral feature into the initial music feature extraction model to obtain a third sample feature vector corresponding to the third sample music clip; Calculating the loss value of the initial music feature extraction model based on the matching label and the distance between the first sample feature vector and the second sample feature vector includes: A loss value of the initial music feature extraction model is calculated based on the matching label, the distance between the first sample feature vector and the second sample feature vector, and the distance between the first sample feature vector and the third sample feature vector.

10. The method according to claim 9, wherein: Before extracting the third spectrum feature of the third sample music segment, the method further includes: Perform data enhancement processing on the third sample music clip.

11. The method according to claim 8, wherein The data enhancement processing includes at least one of the following: adding noise, adding room reverberation effect, performing time shift, and adjusting volume.

12. The method according to any one of claims 7 to 11, further comprising: Get new music from the network according to the preset frequency; as well as The newly added music is used as sample music for training the initial music feature extraction model.

13. A music recognition device, comprising: A first determining module is configured to determine a music segment in the audio to be identified, wherein the music segment is an audio segment containing music components; an extraction module, configured to extract a first feature vector of the music clip; as well as A second determining module is configured to determine at least one target music that matches the audio to be recognized from a music library based on the first feature vector; In response to the audio to be recognized being an audio data stream recorded in real time by a user, the first determining module is further configured to: Sequentially intercepting audio frames of a preset duration from the audio data stream; For each audio frame extracted, detecting whether the audio frame contains a music component; and In response to the number of a plurality of consecutive audio frames containing music components reaching a first threshold, stopping framing and music component detection on subsequent audio data, and determining audio data subsequent to the plurality of audio frames that has not been detected for music components as the music segment, wherein the plurality of audio frames are not cached as the music segment; and In response to the audio to be identified being an audio file recorded and uploaded by a user, the first determining module is further configured to: Dividing the audio file into a plurality of audio frames according to a preset duration, wherein the preset duration for dividing the audio file is greater than a preset duration for intercepting audio frames from the audio data stream; For each of the plurality of audio frames of the audio file, detecting whether the audio frame contains a music component; determining one or more consecutive audio frames containing music components in the audio file as a candidate music segment to obtain at least one candidate music segment corresponding to the audio file, wherein each audio frame contained in each candidate music segment contains music components; and The candidate music segment that includes the largest number of audio frames among the at least one candidate music segment is determined as the music segment.

14. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause a computer to execute the method according to any one of claims 1-12.

16. A computer program product comprising a computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images