Song playing material matching method and device, equipment and medium

Abstract

The song playing material matching method and device, equipment and medium provided by the application relate to the computer technical field.In the application, first, the singing audio data corresponding to the target song and the song identification data are obtained;second, the singing audio data is subjected to potential semantic mining to form an audio semantic vector;then, the song identification data is subjected to potential semantic mining to form an identification semantic vector;further, semantic information having a correlation relationship with the semantic information of the identification semantic vector is mined from the audio semantic vector to form an audio identification correlation vector;finally, based on the audio identification correlation vector, material matching analysis is performed to obtain a material matching probability distribution, and based on the material matching probability distribution, the matched target material is determined.Based on the above, the problem that the reliability of material matching is relatively low in the prior art can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, and medium for matching materials during song playback. Background Technology

[0002] Playing relevant materials (images, videos, etc.) while a user sings a song is a standard practice that can increase user engagement with the platform. However, in current technology, the materials and songs are pre-bound. For example, the binding analysis of materials and songs can be performed manually in advance to determine the materials that match each song. Thus, when a user sings a song, the matching materials can be directly determined and played accordingly. However, this method only considers the information of the song itself, and different users may have different preferences for materials. Therefore, there is a problem of relatively low reliability in material matching. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, apparatus, device and medium for matching materials during song playback, so as to improve the problem of relatively low reliability of material matching in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: A method for matching materials during song playback, including: Acquire the singing audio data and song identification data corresponding to the target song, wherein the singing audio data is formed by collecting the voice of the target user singing the target song, and the song identification data is used to reflect the emotional type and style type of the target song; Latent semantic mining is performed on the singing audio data to form an audio semantic vector; Latent semantic mining is performed on the song identifier data to form an identifier semantic vector; Semantic information that is related to the semantic information of the identifier semantic vector is extracted from the audio semantic vector to form an audio identifier association vector; Based on the audio identifier association vector, a material matching analysis is performed to obtain a material matching probability distribution. Based on the material matching probability distribution, a matching target material is determined. The material matching probability distribution is used to characterize the matching probability of each material in the material library. The target material is used to be played during the process of the target user singing the target song.

[0005] In a preferred embodiment of this application, in the above-mentioned material matching method for song playback, the step of mining semantic information that has a correlation with the semantic information of the identifier semantic vector from the audio semantic vector to form an audio identifier correlation vector includes: From the low-frequency audio semantic vectors included in the audio semantic vector, semantic information that is related to the semantic information of the identifier semantic vector is extracted to form a low-frequency association vector; From the high-frequency audio semantic vectors included in the audio semantic vector, semantic information that has a correlation with the semantic information of the identifier semantic vector is extracted to form a high-frequency correlation vector; The low-frequency correlation vector and the high-frequency correlation vector are fused to form an audio identifier correlation vector.

[0006] In a preferred embodiment of this application, in the above-mentioned material matching method for song playback, the step of mining semantic information that is correlated with the semantic information of the identifier semantic vector from the low-frequency audio semantic vector included in the audio semantic vector to form a low-frequency correlation vector includes: The low-frequency audio semantic vectors included in the audio semantic vectors are segmented at multiple granularities to form a set of segmented semantic vectors corresponding to each granularity. Each set of segmented semantic vectors includes at least one segmented semantic vector, and when multiple segmented semantic vectors are included, the dimensions of the multiple segmented semantic vectors are the same. For each of the segmented semantic vector sets, based on the identifier semantic vector, association semantic mining is performed on each segmented semantic vector in the segmented semantic vector set to form the association semantic mining result corresponding to the segmented semantic vector set; Based on the corresponding granularity relationships, the semantic mining results corresponding to multiple granularities are aggregated to form a low-frequency association vector.

[0007] In a preferred embodiment of this application, in the above-mentioned material matching method for song playback, the step of performing association semantic mining on each segmented semantic vector in the segmented semantic vector set based on the identifier semantic vector to form the association semantic mining result corresponding to the segmented semantic vector set includes: For each of the segmented semantic vector sets, the size of the identifier semantic vector is adjusted based on the size of each segmented semantic vector in the segmented semantic vector set to obtain the size-adjusted identifier semantic vector, and the shared attention parameters between the size-adjusted identifier semantic vector and each segmented semantic vector in the segmented semantic vector set are determined. Based on the shared attention parameters, a weighted summation is performed on each segmentation semantic vector in the segmentation semantic vector set to obtain the attention vector of each segmentation semantic vector; The attention vector of each segmented semantic vector is concatenated to form the associated semantic mining result.

[0008] In a preferred embodiment of this application, in the aforementioned material matching method for song playback, the step of aggregating the semantic mining results corresponding to multiple granularities to form a low-frequency association vector based on the corresponding granularity relationship includes: Following the direction of granularity from large to small, the semantic mining results corresponding to multiple granularities are sequentially subjected to pooling operations of different scales to form multiple first pooling results. Among them, the semantic mining results corresponding to multiple granularities have the same size. Following the direction of increasing granularity, the semantic mining results corresponding to multiple granularities are sequentially subjected to pooling operations of different scales to form multiple second pooling results; The first and second pooling results corresponding to pooling operations of the same scale are combined to form a corresponding pooling result combination. For each pooling result combination, the first pooling result and the second pooling result included in the pooling result combination are subjected to bidirectional cross-attention fusion to form the cross-attention fusion result corresponding to the pooling result combination; Multiple cross-attention fusion results corresponding to various granularities are added together to form a low-frequency correlation vector.

[0009] In a preferred embodiment of this application, the step of performing latent semantic mining on the vocal audio data to form an audio semantic vector in the aforementioned material matching method for song playback includes: The singing audio data is subjected to Fourier transform to form an audio spectrogram; The high-frequency information of the audio spectrogram is hidden to form a low-frequency spectrogram; The low-frequency information of the audio spectrogram is hidden to form a high-frequency spectrogram, wherein the union of the information of the low-frequency spectrogram and the high-frequency spectrogram is the audio spectrogram; The low-frequency spectrogram is subjected to latent semantic mining to form a low-frequency audio semantic vector; Latent semantic mining is performed on the high-frequency spectrogram to form a high-frequency audio semantic vector, wherein the audio semantic vector includes the low-frequency audio semantic vector and the high-frequency audio semantic vector.

[0010] In a preferred embodiment of this application, the step of performing latent semantic mining on the low-frequency spectrogram to form a low-frequency audio semantic vector in the aforementioned material matching method for song playback includes: Obtain the reference low-frequency spectrum corresponding to the reference singing audio data of the target song, wherein the reference singing audio data is formed by collecting the sound of other users singing the target song in a noise-free environment; The reference low-frequency spectrum and the low-frequency spectrum are convolved respectively to form a first spectral convolution vector corresponding to the reference low-frequency spectrum and a second spectral convolution vector corresponding to the low-frequency spectrum; Based on the first spectral convolution vector, spectral gating parameters for gating the second spectral convolution vector are determined. Based on the aforementioned spectrum gating parameters, the second spectrum convolution vector is gating and filtering to form a low-frequency audio semantic vector.

[0011] This application also provides a material matching device for song playback, including: The song data acquisition module is used to acquire the singing audio data and song identification data corresponding to the target song. The singing audio data is formed by collecting the voice of the target user singing the target song, and the song identification data is used to reflect the emotional type and style type of the target song. The audio semantic mining module is used to perform latent semantic mining on the singing audio data to form an audio semantic vector; The identifier semantic mining module is used to perform latent semantic mining on the song identifier data to form an identifier semantic vector; The semantic association mining module is used to mine semantic information that is related to the semantic information of the identifier semantic vector from the audio semantic vector, and form an audio identifier association vector; The material matching module is used to perform material matching analysis based on the audio identifier association vector to obtain the material matching probability distribution, and to determine the target material to be matched based on the material matching probability distribution. The material matching probability distribution is used to characterize the matching probability of each material in the material library, and the target material is used to be played during the process of the target user singing the target song.

[0012] Based on the above, this application also provides an electronic device, including: Memory, used to store computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the above-described method for matching materials during song playback.

[0013] Based on the above, this application also provides a computer-readable storage medium storing a computer program that, when executed, performs the various steps of the above-described method for matching materials during song playback.

[0014] The method, apparatus, device, and medium for matching materials during song playback provided in this application firstly acquire the singing audio data and song identification data corresponding to the target song; secondly, perform latent semantic mining on the singing audio data to form an audio semantic vector; then, perform latent semantic mining on the song identification data to form an identification semantic vector; further, mine semantic information that has a correlation with the semantic information of the identification semantic vector from the audio semantic vector to form an audio identification association vector; finally, perform material matching analysis based on the audio identification association vector to obtain a material matching probability distribution, and determine the target material to be matched based on the material matching probability distribution. Based on the above, on the one hand, because the semantic information in the voice of the target user singing the target song is mined, the target material determined based on this has a higher degree of fit with the target user. On the other hand, because the semantic information of the emotional type and style type of the target song is also mined to extract related semantic information, the resulting audio identifier association vector not only represents the relevant semantic information of the target user, but also focuses on the representation of emotions and styles, thereby further improving the fit of the determined target material. Therefore, it can improve the problem of relatively low reliability of material matching in the existing technology and increase the stickiness of the target user to the corresponding platform. Attached Figure Description

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0016] Figure 1 A structural block diagram of an electronic device provided in an embodiment of this application.

[0017] Figure 2 This is a flowchart illustrating the material matching method for song playback provided in an embodiment of this application.

[0018] Figure 3 This is a schematic diagram illustrating the latent semantic mining of audio data provided in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram illustrating the aggregation of semantic mining results corresponding to multiple granularities, as provided in an embodiment of this application.

[0020] Figure 5 A block diagram illustrating a material matching device for song playback provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] like Figure 1 As shown in the figure, this application provides an electronic device. The electronic device may include a memory, a processor, and a material matching device for song playback.

[0024] In detail, the memory and the processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, the memory and the processor can be electrically connected via one or more communication buses or signal lines. The material matching device for song playback includes at least one software functional module stored in the memory in the form of software or firmware. The processor is used to execute the executable computer program stored in the memory, such as the software functional module and computer program included in the material matching device for song playback, to implement the material matching method for song playback provided in this application embodiment.

[0025] Optionally, the memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0026] Furthermore, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0027] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown may include, for example, a communication unit for exchanging information with other devices, such as an audio acquisition unit.

[0028] Combination Figure 2 This application also provides a material matching method applicable to song playback in the aforementioned electronic device. The method steps defined in the process related to the material matching method during song playback can be implemented by the electronic device.

[0029] The following will be about Figure 2 The specific process shown will be explained in detail.

[0030] Step S110: Obtain the singing audio data and song identification data corresponding to the target song.

[0031] In this embodiment, the electronic device can acquire the vocal audio data and song identification data corresponding to the target song. The vocal audio data is formed by capturing the voice of a target user singing the target song, for example, through audio acquisition devices such as microphones. Furthermore, the song identification data reflects the emotional type and style of the target song, and can be pre-labeled. The emotional type can include happiness, sadness, tension, anxiety, anger, surprise, loneliness, etc., and the style can include classical, blues, hip-hop, pop, rock, jazz, reggae, metal, disco, and country, etc.

[0032] Step S120: Perform latent semantic mining on the singing audio data to form an audio semantic vector.

[0033] In this embodiment of the application, after acquiring the singing audio data, the electronic device can perform latent semantic mining on the singing audio data to form an audio semantic vector. That is, it mines the latent semantic information in the singing audio data and represents it in vector form.

[0034] Step S130: Perform latent semantic mining on the song identifier data to form an identifier semantic vector.

[0035] In this embodiment of the application, after obtaining the song identification data, the electronic device can perform latent semantic mining on the song identification data to form an identification semantic vector. That is, it mines the latent semantic information in the song identification data and represents it in vector form.

[0036] Step S140: Extract semantic information that is related to the semantic information of the identifier semantic vector from the audio semantic vector to form an audio identifier association vector.

[0037] In this embodiment, after mining the audio semantic vector and the identifier semantic vector, the electronic device can extract semantic information from the audio semantic vector that is related to the semantic information of the identifier semantic vector, forming an audio identifier association vector. That is, since the audio semantic vector represents the global semantic information (which may also include noise) in the singing audio data, its semantic representation accuracy is relatively low for application scenarios such as material matching. Therefore, association mining can be performed by combining the semantic information of the identifier semantic vector, allowing the extraction of semantic information related to emotion and style from the global semantic information represented by the audio semantic vector. Since emotion and style have a positive and active representation effect on the user's fit, the resulting audio identifier association vector can have high semantic representation accuracy in application scenarios such as material matching.

[0038] Step S150: Perform material matching analysis based on the audio identifier association vector to obtain the material matching probability distribution, and determine the target material to be matched based on the material matching probability distribution.

[0039] In this embodiment, after forming the audio identifier association vector, the electronic device can perform material matching analysis based on the audio identifier association vector to obtain a material matching probability distribution, and determine the target material to be matched based on the material matching probability distribution. The material matching probability distribution is used to characterize the matching probability of each material in the material library, and the target material can be the material corresponding to the matching probability with the maximum value among all matching probabilities. Furthermore, the target material is used to be played during the process of the target user singing the target song.

[0040] Based on the above, on the one hand, because the semantic information in the voice of the target user singing the target song is mined, the target material determined based on this has a higher degree of fit with the target user. On the other hand, because the semantic information of the emotional type and style type of the target song is also mined to extract related semantic information, the resulting audio identifier association vector not only represents the relevant semantic information of the target user, but also focuses on the representation of emotions and styles, thereby further improving the fit of the determined target material. Therefore, it can improve the problem of relatively low reliability of material matching in the existing technology and increase the stickiness of the target user to the corresponding platform.

[0041] Firstly, regarding step S110, it should be noted that the specific method for obtaining the singing audio data and song identification data corresponding to the target song is not limited and can be selected according to actual needs.

[0042] For example, in an alternative implementation, audio data can be directly and in real-time acquired from the audio acquisition device that captures the sound of the target user singing the target song, thereby obtaining the singing audio data.

[0043] For example, in another alternative implementation, the audio data collected by the audio acquisition device from the target user singing the target song can be preprocessed (such as noise removal), and then the preprocessed audio data can be sent to the electronic device to obtain the singing audio data.

[0044] Secondly, it should be noted that the specific method for performing latent semantic mining on the singing audio data is not limited and can be selected according to actual needs.

[0045] For example, in an alternative implementation, semantic mining can be performed directly on the time-domain singing audio data to form a corresponding audio semantic vector. The specific mining process can be referred to the relevant existing technologies, which will not be elaborated here.

[0046] For example, in another alternative implementation, in order to improve the semantic representation accuracy of the mined audio semantic vector, the singing audio data can be mined with high precision and fine granularity. Specifically, the above step S120 can further include steps S121, S122, S123, S124 and S125, the specific contents of each step are as follows.

[0047] Step S121: Perform Fourier transform on the singing audio data to form an audio spectrogram.

[0048] In this embodiment, the singing audio data can be Fourier transformed to form an audio spectrogram. That is, the time-domain singing audio data can be converted to the frequency domain to form a corresponding audio spectrogram, such as... Figure 3 As shown.

[0049] Step S122: Hide the high-frequency information in the audio spectrogram to form a low-frequency spectrogram.

[0050] In this embodiment of the application, after the audio spectrogram is formed, the high-frequency information of the audio spectrogram can be hidden to form a low-frequency spectrogram. For example, the parameter value of the region corresponding to the high-frequency information in the audio spectrogram can be configured to 0 to hide the high-frequency information.

[0051] Step S123: Hide the low-frequency information in the audio spectrogram to form a high-frequency spectrogram.

[0052] In this embodiment, after forming the audio spectrogram, low-frequency information can be hidden to form a high-frequency spectrogram. For example, the parameter values ​​of the regions corresponding to low-frequency information in the audio spectrogram can be configured to 0 to hide the low-frequency information. The audio spectrogram is the union of the information in the low-frequency spectrogram and the high-frequency spectrogram. For example, low frequency can refer to frequencies less than or equal to 200Hz, and high frequency can refer to frequencies greater than 200Hz; that is, 200Hz is used to distinguish between low and high frequencies.

[0053] Step S124: Perform latent semantic mining on the low-frequency spectrogram to form a low-frequency audio semantic vector.

[0054] In this embodiment of the application, after forming the low-frequency spectrogram, latent semantic mining can be performed on the low-frequency spectrogram to form a low-frequency audio semantic vector. That is, latent semantic information in the low-frequency spectrogram can be mined and represented by a vector.

[0055] Step S125: Perform latent semantic mining on the high-frequency spectrogram to form a high-frequency audio semantic vector.

[0056] In this embodiment, after forming the high-frequency spectrogram, latent semantic mining can be performed on the high-frequency spectrogram to form a high-frequency audio semantic vector. That is, latent semantic information can be mined from the high-frequency spectrogram and represented by a vector. The audio semantic vector includes both low-frequency and high-frequency audio semantic vectors. Based on this, high-frequency and low-frequency semantic information can be mined separately, avoiding mutual interference, thereby improving the accuracy of latent semantic mining and ensuring the reliability of the obtained audio semantic vector.

[0057] It is understood that the specific method of performing latent semantic mining on the low-frequency spectrum in step S124 above is not limited. For example, in an alternative implementation, in order to further improve the accuracy of latent semantic mining, step S124 above may further include steps S124a, S124b, S124c and S124d, the specific contents of each step are as follows.

[0058] Step S124a: Obtain the reference low-frequency spectrum corresponding to the reference singing audio data of the target song.

[0059] In this embodiment, a reference low-frequency spectrogram corresponding to the reference singing audio data of the target song can be obtained. The reference singing audio data is generated by collecting the sound of other users singing the target song in a noise-free environment. Then, the reference singing audio data can be subjected to a Fourier transform to obtain the corresponding reference spectrogram. Finally, high-frequency information is hidden from the reference spectrogram to form a reference low-frequency spectrogram.

[0060] Step S124b: Convolve the reference low-frequency spectrum map and the low-frequency spectrum map respectively to form a first spectral convolution vector corresponding to the reference low-frequency spectrum map and a second spectral convolution vector corresponding to the low-frequency spectrum map.

[0061] In this embodiment of the application, after forming the reference low-frequency spectrum, the reference low-frequency spectrum and the low-frequency spectrum can be convolved to form a first spectral convolution vector corresponding to the reference low-frequency spectrum and a second spectral convolution vector corresponding to the low-frequency spectrum. That is, the reference low-frequency spectrum is convolved to form the first spectral convolution vector, and the low-frequency spectrum is convolved to form the second spectral convolution vector.

[0062] Step S124c: Determine the spectrum gating parameters for gating the second spectrum convolution vector based on the first spectrum convolution vector.

[0063] In this embodiment, after forming the first spectral convolution vector and the second spectral convolution vector, spectral gating parameters for gating the second spectral convolution vector can be determined based on the first spectral convolution vector. For example, the first spectral convolution vector can first be linearly mapped (i.e., Y = Ax + b, where Y is the result of the linear mapping, x is the first spectral convolution vector, A is the weight parameter of the linear mapping, and b is the bias parameter of the linear mapping). Then, the result of the linear mapping can be non-linearly activated, such as through functions like sigmoid, so that each value in the spectral gating parameters belongs to 0-1. This can then be used to characterize the importance of parameters at corresponding positions in the second spectral convolution vector.

[0064] Step S124d: Based on the spectrum gating and filtering parameters, the second spectrum convolution vector is gating and filtering to form a low-frequency audio semantic vector.

[0065] In this embodiment, after forming the spectrum gating parameters, the second spectrum convolution vector can be gated and filtered based on the spectrum gating parameters to form a low-frequency audio semantic vector. For example, since the spectrum gating parameters can be used to characterize the importance of parameters at corresponding positions in the second spectrum convolution vector, the spectrum gating parameters and the second spectrum convolution vector can be multiplied bitwise to form a low-frequency audio semantic vector, thus achieving a focused representation of important semantic information.

[0066] It is understood that in step S125 above, the specific method of performing latent semantic mining on the high-frequency spectrogram is not limited. The processing logic in steps S124a, S124b, S124c and S124d above can be referred to to perform latent semantic mining on the high-frequency spectrogram, that is, to combine the reference high-frequency spectrogram corresponding to the reference singing audio data corresponding to the target song.

[0067] Thirdly, regarding step S130, it should be noted that the specific method for performing latent semantic mining on the song identifier data is not limited and can be selected according to actual needs.

[0068] For example, in an alternative implementation, the song identification data can be text data. Therefore, any existing technique for semantic mining of text can be used, without specific limitations. For example, the song identification data can be embedded based on a trained word embedding model to obtain the identification semantic vector.

[0069] Fourthly, regarding step S140, it should be noted that the specific method for mining semantic information that is related to the semantic information of the identifier semantic vector from the audio semantic vector is not limited and can be selected according to actual needs.

[0070] For example, in an alternative implementation, the audio semantic vector can be cross-attention processed based on the identifier semantic vector to form an audio identifier association vector.

[0071] For example, in another alternative implementation, as mentioned above, the audio semantic vector may include low-frequency audio semantic vectors and high-frequency audio semantic vectors, that is, formed by mining low-frequency semantic information and high-frequency semantic information respectively. Based on this, the above-mentioned step S140 may further include steps S141, S142 and S143, the specific contents of each step are as follows.

[0072] Step S141: Extract semantic information that is related to the semantic information of the identifier semantic vector from the low-frequency audio semantic vector included in the audio semantic vector, and form a low-frequency association vector.

[0073] In this embodiment of the application, semantic information that is related to the semantic information of the identifier semantic vector can be extracted from the low-frequency audio semantic vector included in the audio semantic vector to form a low-frequency association vector.

[0074] Step S142: Extract semantic information that is related to the semantic information of the identifier semantic vector from the high-frequency audio semantic vector included in the audio semantic vector, and form a high-frequency correlation vector.

[0075] In this embodiment, semantic information that is correlated with the semantic information of the identifier semantic vector can be mined from the high-frequency audio semantic vectors included in the audio semantic vector to form a high-frequency correlation vector. It should be noted that the method for mining semantic information correlated with the semantic information of the identifier semantic vector from the high-frequency audio semantic vector can be the same as the method for mining semantic information correlated with the semantic information of the identifier semantic vector from the low-frequency audio semantic vector; that is, the processing logic is the same.

[0076] Step S143: Merge the low-frequency association vector and the high-frequency association vector to form an audio identifier association vector.

[0077] In this embodiment, after forming low-frequency and high-frequency association vectors respectively, the low-frequency and high-frequency association vectors can be fused to form an audio identifier association vector. That is, semantic information related to associations can be mined separately from both high-frequency and low-frequency dimensions, resulting in finer granularity and higher accuracy, while avoiding interference between high-frequency and low-frequency information. Thus, after obtaining more accurate low-frequency and high-frequency association vectors, the semantic information from the low-frequency and high-frequency frequencies can be fused to obtain an audio identifier association vector with both high accuracy and semantic richness. For example, the low-frequency and high-frequency association vectors can be concatenated and then subjected to convolution, pooling, and fully connected processing to form the audio identifier association vector; alternatively, the low-frequency and high-frequency association vectors can be added or averaged to form the audio identifier association vector.

[0078] It is understood that in step S141 above, the specific method of mining semantic information that has a correlation with the semantic information of the identifier semantic vector is not limited. For example, in an alternative implementation, in order to further improve the accuracy of semantic information mining, step S141 above may further include steps S141a, S141b and S141c, the specific contents of each step are as follows.

[0079] Step S141a: The low-frequency audio semantic vector included in the audio semantic vector is segmented at multiple granularities to form a set of segmented semantic vectors corresponding to each granularity.

[0080] In this embodiment, the low-frequency audio semantic vector included in the audio semantic vector can be segmented at various granularities to form a set of segmented semantic vectors corresponding to each granularity. Each set of segmented semantic vectors includes at least one segmented semantic vector, and when multiple segmented semantic vectors are included, the dimensions of the multiple segmented semantic vectors are the same (segmented semantic vectors formed by segmentation at different granularities have different dimensions; for example, the more segmented semantic vectors formed, the smaller the size of the segmented semantic vectors; furthermore, when the set of segmented semantic vectors includes only one segmented semantic vector, the size of that segmented semantic vector is equal to the size of the low-frequency audio semantic vector). Additionally, for any set of segmented semantic vectors, the at least one segmented semantic vector included in the set can be concatenated to form the low-frequency audio semantic vector.

[0081] Step S141b: For each of the segmented semantic vector sets, based on the identifier semantic vector, perform association semantic mining on each segmented semantic vector in the segmented semantic vector set to form the association semantic mining result corresponding to the segmented semantic vector set.

[0082] In this embodiment, after forming a set of segmented semantic vectors corresponding to each granularity, for each set of segmented semantic vectors, based on the identifier semantic vector, association semantic mining is performed on each segmented semantic vector in the set to form an association semantic mining result corresponding to that set of segmented semantic vectors. Thus, because the granularity and size of the segmented semantic vectors in different sets of segmented semantic vectors are different, association semantic mining can be performed at different granularities and sizes, thereby achieving multi-granularity and multi-size mining, resulting in more thorough mining.

[0083] Step S141c: Based on the corresponding granularity relationship, aggregate the semantic mining results corresponding to multiple granularities to form a low-frequency association vector.

[0084] In this embodiment, after generating the semantic mining results, the semantic mining results corresponding to multiple granularities can be aggregated based on the corresponding granularity relationships to form a low-frequency association vector. That is, during aggregation, the corresponding granularity needs to be considered to improve the aggregation effect and thus generate a low-frequency association vector with higher semantic representation accuracy.

[0085] It is understood that in step S141b above, the specific method of performing association semantic mining on each segmented semantic vector in the segmented semantic vector set is not limited. For example, in an alternative implementation, in order to improve the efficiency of association semantic mining and reduce computational overhead, step S141b above may further include steps b1, b2 and b3, the specific contents of each step are as follows.

[0086] Step b1: For each of the segmented semantic vector sets, the size of the identifier semantic vector is adjusted based on the size of each segmented semantic vector in the segmented semantic vector set to obtain the size-adjusted identifier semantic vector, and the shared attention parameter between the size-adjusted identifier semantic vector and each segmented semantic vector in the segmented semantic vector set is determined.

[0087] In this embodiment, considering that segmentation semantic vectors of different granularities generally have different sizes, and therefore, the sizes of the identifier semantic vectors are also different, for each set of segmentation semantic vectors, the size of the identifier semantic vector is adjusted based on the size of each segmentation semantic vector in the set, to obtain a size-adjusted identifier semantic vector (with the same size as the segmentation semantic vector). Furthermore, the shared attention parameters between the size-adjusted identifier semantic vector and each segmentation semantic vector in the set are determined. For example, the mean vector (or cluster center vector) of each segmentation semantic vector can be determined. Then, the size-adjusted identifier semantic vector and the transpose of the mean vector can be multiplied to obtain the corresponding shared attention parameters (a matrix composed of multiple parameters).

[0088] Step b2: Based on the shared attention parameters, perform a weighted summation calculation on each segmentation semantic vector in the segmentation semantic vector set to obtain the attention vector of each segmentation semantic vector.

[0089] In this embodiment, after forming the shared attention parameters, a weighted summation can be performed on each segmented semantic vector in the segmented semantic vector set based on the shared attention parameters to obtain the attention vector for each segmented semantic vector. Thus, since the shared attention parameters can represent the overall correlation between each segmented semantic vector in the segmented semantic vector set, multiple attention parameters (i.e., the attention parameters that are associated with each segmented semantic vector) can be replaced by the shared attention parameters, thereby reducing the resource consumption of attention parameter calculation and improving computational efficiency. Based on this, after performing a weighted summation calculation on the shared attention parameters and each segmented semantic vector, the attention vector for each segmented semantic vector can be obtained.

[0090] Step b3: Concatenate the attention vectors of each segmented semantic vector to form the associated semantic mining result.

[0091] In the embodiments of this application, after forming the attention vector of each segmented semantic vector, the attention vectors of each segmented semantic vector can be concatenated to form the associated semantic mining result.

[0092] It is understandable that in step S141c above, the specific method of aggregating the association semantic mining results corresponding to multiple granularities is not limited. For example, in an alternative implementation, in order to mine association semantic information from association semantic mining results of different granularities, thereby extracting more complex semantic information and making the formed low-frequency association vector have better semantic representation capabilities, step S141c above can further include steps c1, c2, c3, c4, and c5. The specific contents of each step are as follows (in conjunction with...). Figure 4 ).

[0093] Step c1: Following the direction of granularity from large to small, perform pooling operations of different scales on the associated semantic mining results corresponding to multiple granularities in sequence to form multiple first pooling results.

[0094] In this embodiment, pooling operations of different scales can be performed sequentially on the semantic mining results corresponding to multiple granularities, from large to small, to form multiple first pooling results. For example, the size of the first pooling results gradually decreases from large to small granularity. The semantic mining results corresponding to multiple granularities have the same size.

[0095] Step c2: Following the direction of increasing granularity, perform pooling operations of different scales on the semantic mining results corresponding to various granularities in sequence to form multiple second pooling results.

[0096] In this embodiment of the application, pooling operations of different scales can be performed sequentially on the associated semantic mining results corresponding to multiple granularities in the direction of increasing granularity to form multiple second pooling results. For example, the size of the second pooling results formed in the direction of increasing granularity gradually decreases. That is to say, the size of the first pooling result corresponding to the associated semantic mining result with the largest granularity is the largest, and the size of the second pooling result corresponding to the associated semantic mining result with the largest granularity is the smallest.

[0097] Step c3: Combine the first pooling result and the second pooling result corresponding to pooling operations with the same scale to form a corresponding pooling result combination.

[0098] In this embodiment, after forming the first pooling result and the second pooling result, the first pooling result and the second pooling result corresponding to pooling operations of the same scale can be combined to form a corresponding pooling result combination. For example, the first pooling result corresponding to the largest granularity association semantic mining result and the second pooling result corresponding to the smallest granularity association semantic mining result can be combined to form a corresponding pooling result combination, that is, to achieve a combination of different granularities.

[0099] Step c4: For each pooling result combination, perform bidirectional cross-attention fusion on the first pooling result and the second pooling result included in the pooling result combination to form the cross-attention fusion result corresponding to the pooling result combination.

[0100] In this embodiment, after forming the pooling result combination, for each pooling result combination, a bidirectional cross-attention fusion is performed on the first pooling result and the second pooling result included in the pooling result combination to form a cross-attention fusion result corresponding to the pooling result combination. For example, cross-attention processing can be performed on the second pooling result based on the first pooling result to obtain a first cross-attention result, and cross-attention processing can be performed on the first pooling result based on the second pooling result to obtain a second cross-attention result. Furthermore, the first cross-attention result and the second cross-attention result can be averaged or added together to obtain a cross-attention fusion result, thereby achieving a full fusion of the first pooling result and the second pooling result.

[0101] Step c5 involves adding the results of multiple cross-attention fusions corresponding to various granularities to form a low-frequency correlation vector.

[0102] In this embodiment of the application, after the cross-attention fusion result is formed, multiple cross-attention fusion results corresponding to different granularities can be added together to form a low-frequency correlation vector.

[0103] Fifthly, regarding step S150, it should be noted that the specific method for determining the matching target material is not limited and can be selected according to actual needs.

[0104] For example, in an alternative implementation, the audio identifier association vector can be fully connected to form a fully connected vector of size 1*n, where n equals the number of each material in the material library. Then, this fully connected vector can be activated, for example using a softmax function, to obtain a material matching probability distribution. The number of matching probabilities in this distribution is also equal to n, representing the matching probability of each material. Finally, the material with the highest matching probability can be selected as the target material.

[0105] Combination Figure 5 This application also provides a material matching device applicable to song playback in the aforementioned electronic device. The material matching device may include a song data acquisition module, an audio semantic mining module, an identifier semantic mining module, an association semantic mining module, and a material matching module.

[0106] Specifically, the song data acquisition module can be used to acquire the vocal audio data and song identification data corresponding to the target song. The vocal audio data is generated by collecting the voice of a target user singing the target song, and the song identification data reflects the emotional type and style of the target song. In this embodiment, the song data acquisition module can be used to execute... Figure 2 The relevant content regarding the song data acquisition module in step S110 shown can be found in the previous description of step S110.

[0107] Specifically, the audio semantic mining module can be used to perform latent semantic mining on the singing audio data to form an audio semantic vector. In this embodiment, the audio semantic mining module can be used to perform... Figure 2 The relevant content regarding the audio semantic mining module in step S120 shown can be found in the previous description of step S120.

[0108] Specifically, the identifier semantic mining module can be used to perform latent semantic mining on the song identifier data to form an identifier semantic vector. In this embodiment, the identifier semantic mining module can be used to perform... Figure 2 The relevant content regarding the identifier semantic mining module in step S130 shown can be found in the previous description of step S130.

[0109] Specifically, the associated semantic mining module can be used to mine semantic information from the audio semantic vector that has a correlation with the semantic information of the identifier semantic vector, forming an audio identifier association vector. In this embodiment, the associated semantic mining module can be used to perform... Figure 2 The relevant content regarding the associated semantic mining module in step S140 can be found in the preceding description of step S140.

[0110] In detail, the material matching module can be used to perform material matching analysis based on the audio identifier association vector to obtain a material matching probability distribution, and, based on the material matching probability distribution, determine the target material to be matched. The material matching probability distribution is used to characterize the matching probability of each material in the material library, and the target material is used to be played during the process of the target user singing the target song. In this embodiment, the material matching module can be used to perform... Figure 2 The relevant content regarding the material matching module in step S150 shown can be found in the previous description of step S150.

[0111] In this embodiment of the application, corresponding to the above-described material matching method for song playback applied to the electronic device, a computer-readable storage medium is also provided. This computer-readable storage medium stores a computer program, which, when executed, performs each step of the material matching method for song playback. The steps executed by the aforementioned computer program are not described in detail here, but can be referred to the preceding explanation of the material matching method for song playback.

[0112] In summary, the song playback material matching method, apparatus, device, and medium provided in this application firstly acquire the singing audio data and song identification data corresponding to the target song; secondly, perform latent semantic mining on the singing audio data to form an audio semantic vector; then, perform latent semantic mining on the song identification data to form an identification semantic vector; further, mine semantic information that has a correlation with the semantic information of the identification semantic vector from the audio semantic vector to form an audio identification association vector; finally, perform material matching analysis based on the audio identification association vector to obtain the material matching probability distribution, and determine the target material to be matched based on the material matching probability distribution. Based on the above, on the one hand, because the semantic information in the voice of the target user singing the target song is mined, the target material determined based on this has a higher degree of fit with the target user. On the other hand, because the semantic information of the emotional type and style type of the target song is also mined to extract related semantic information, the resulting audio identifier association vector not only represents the relevant semantic information of the target user, but also focuses on the representation of emotions and styles, thereby further improving the fit of the determined target material. Therefore, it can improve the problem of relatively low reliability of material matching in the existing technology and increase the stickiness of the target user to the corresponding platform.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0114] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0115] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for matching materials during song playback, characterized in that, include: Acquire the singing audio data and song identification data corresponding to the target song, wherein the singing audio data is formed by collecting the voice of the target user singing the target song, and the song identification data is used to reflect the emotional type and style type of the target song; Latent semantic mining is performed on the singing audio data to form an audio semantic vector; Latent semantic mining is performed on the song identifier data to form an identifier semantic vector; Semantic information that is related to the semantic information of the identifier semantic vector is extracted from the audio semantic vector to form an audio identifier association vector; Based on the audio identifier association vector, a material matching analysis is performed to obtain a material matching probability distribution. Based on the material matching probability distribution, a matching target material is determined. The material matching probability distribution is used to characterize the matching probability of each material in the material library. The target material is used to be played during the process of the target user singing the target song.

2. The method for matching materials during song playback according to claim 1, characterized in that, The step of mining semantic information that is related to the semantic information of the identifier semantic vector from the audio semantic vector to form an audio identifier association vector includes: From the low-frequency audio semantic vectors included in the audio semantic vector, semantic information that is related to the semantic information of the identifier semantic vector is extracted to form a low-frequency association vector; From the high-frequency audio semantic vectors included in the audio semantic vector, semantic information that has a correlation with the semantic information of the identifier semantic vector is extracted to form a high-frequency correlation vector; The low-frequency correlation vector and the high-frequency correlation vector are fused to form an audio identifier correlation vector.

3. The material matching method for song playback according to claim 2, characterized in that, The step of mining semantic information that is correlated with the semantic information of the identifier semantic vector from the low-frequency audio semantic vector included in the audio semantic vector, and forming a low-frequency correlation vector, includes: The low-frequency audio semantic vectors included in the audio semantic vectors are segmented at multiple granularities to form a set of segmented semantic vectors corresponding to each granularity. Each set of segmented semantic vectors includes at least one segmented semantic vector, and when multiple segmented semantic vectors are included, the dimensions of the multiple segmented semantic vectors are the same. For each of the segmented semantic vector sets, based on the identifier semantic vector, association semantic mining is performed on each segmented semantic vector in the segmented semantic vector set to form the association semantic mining result corresponding to the segmented semantic vector set; Based on the corresponding granularity relationships, the semantic mining results corresponding to multiple granularities are aggregated to form a low-frequency association vector.

4. The material matching method for song playback according to claim 3, characterized in that, The step of performing association semantic mining on each segmented semantic vector in each segmented semantic vector set based on the identifier semantic vector to form the association semantic mining result corresponding to the segmented semantic vector set includes: For each of the segmented semantic vector sets, the size of the identifier semantic vector is adjusted based on the size of each segmented semantic vector in the segmented semantic vector set to obtain the size-adjusted identifier semantic vector, and the shared attention parameters between the size-adjusted identifier semantic vector and each segmented semantic vector in the segmented semantic vector set are determined. Based on the shared attention parameters, a weighted summation is performed on each segmentation semantic vector in the segmentation semantic vector set to obtain the attention vector of each segmentation semantic vector; The attention vector of each segmented semantic vector is concatenated to form the associated semantic mining result.

5. The method for matching materials during song playback according to claim 3, characterized in that, The step of aggregating the semantic mining results corresponding to multiple granularities based on the corresponding granularity relationships to form a low-frequency association vector includes: Following the direction of granularity from large to small, the semantic mining results corresponding to multiple granularities are sequentially subjected to pooling operations of different scales to form multiple first pooling results. Among them, the semantic mining results corresponding to multiple granularities have the same size. Following the direction of increasing granularity, the semantic mining results corresponding to multiple granularities are sequentially subjected to pooling operations of different scales to form multiple second pooling results; The first and second pooling results corresponding to pooling operations of the same scale are combined to form a corresponding pooling result combination. For each pooling result combination, the first pooling result and the second pooling result included in the pooling result combination are subjected to bidirectional cross-attention fusion to form the cross-attention fusion result corresponding to the pooling result combination; Multiple cross-attention fusion results corresponding to various granularities are added together to form a low-frequency correlation vector.

6. The method for matching materials during song playback according to any one of claims 1-5, characterized in that, The step of performing latent semantic mining on the singing audio data to form an audio semantic vector includes: The singing audio data is subjected to Fourier transform to form an audio spectrogram; The high-frequency information of the audio spectrogram is hidden to form a low-frequency spectrogram; The low-frequency information of the audio spectrogram is hidden to form a high-frequency spectrogram, wherein the union of the information of the low-frequency spectrogram and the high-frequency spectrogram is the audio spectrogram; The low-frequency spectrogram is subjected to latent semantic mining to form a low-frequency audio semantic vector; Latent semantic mining is performed on the high-frequency spectrogram to form a high-frequency audio semantic vector, wherein the audio semantic vector includes the low-frequency audio semantic vector and the high-frequency audio semantic vector.

7. The material matching method for song playback according to claim 6, characterized in that, The step of performing latent semantic mining on the low-frequency spectrogram to form a low-frequency audio semantic vector includes: Obtain the reference low-frequency spectrum corresponding to the reference singing audio data of the target song, wherein the reference singing audio data is formed by collecting the sound of other users singing the target song in a noise-free environment; The reference low-frequency spectrum and the low-frequency spectrum are convolved respectively to form a first spectral convolution vector corresponding to the reference low-frequency spectrum and a second spectral convolution vector corresponding to the low-frequency spectrum; Based on the first spectral convolution vector, spectral gating parameters for gating the second spectral convolution vector are determined. Based on the aforementioned spectrum gating parameters, the second spectrum convolution vector is gating and filtering to form a low-frequency audio semantic vector.

8. A material matching device for song playback, characterized in that, include: The song data acquisition module is used to acquire the singing audio data and song identification data corresponding to the target song. The singing audio data is formed by collecting the voice of the target user singing the target song, and the song identification data is used to reflect the emotional type and style type of the target song. The audio semantic mining module is used to perform latent semantic mining on the singing audio data to form an audio semantic vector; The identifier semantic mining module is used to perform latent semantic mining on the song identifier data to form an identifier semantic vector; The semantic association mining module is used to mine semantic information that is related to the semantic information of the identifier semantic vector from the audio semantic vector, and form an audio identifier association vector; The material matching module is used to perform material matching analysis based on the audio identifier association vector to obtain the material matching probability distribution, and to determine the target material to be matched based on the material matching probability distribution. The material matching probability distribution is used to characterize the matching probability of each material in the material library, and the target material is used to be played during the process of the target user singing the target song.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor connected to the memory is used to execute a computer program stored in the memory to implement the material matching method for song playback as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a computer program that, when executed, performs the material matching method for song playback as described in any one of claims 1-7.

Images