A method and system for fusion of heterogeneous music data based on symbolic semantics

Abstract

The present invention discloses a heterogeneous music data fusion method and system based on symbolic semantics, which belongs to the field of heterogeneous data fusion and music informatics analysis. Heterogeneous music data acquisition and feature extraction, after data preprocessing, the basic information of the heterogeneous music data and the unique features of each data structure are obtained, and the associated heterogeneous music data and its unified annotation sequence are obtained; a symbolic semantic parsing model is defined, and the symbolic semantic parsing model is used to obtain a stable symbolic semantic observation space from the associated heterogeneous music data and its unified annotation sequence, and significant semantic attributes and controllable semantic constraints are obtained from the stable symbolic semantic observation space; according to the adjustment of the significant semantic attributes and controllable semantic constraints, the adapted data that meets the specific semantic constraints is restored or generated. The present invention can obtain semantic attributes of different dimensions from the semantic parsing process and constrain them, so as to realize interpretable and controllable music data fusion and generation.

Description

Technical Field

[0001] The present invention relates to the field of heterogeneous data fusion and music informatics analysis, and in particular to a method and system for fusion of heterogeneous music data based on symbolic semantics. Background Art

[0002] In recent years, streaming media technology has developed rapidly in the Internet era. Music data in different compilation and storage methods on the Internet has brought convenience to the creation and use of publishers, composers, and Internet users. However, the semantic gap caused by the heterogeneity of music data is also a problem that hinders data interoperability and the development of the creative ecology. Without appropriate heterogeneous fusion technology, the complex and redundant music data on the Internet will bring people the burden of data analysis, music semantic interpretation and quality evaluation, and the original creative ecology will not be able to continue to develop.

[0003] The heterogeneity of music data mainly comes from two aspects. On the one hand, music itself has a complex and diverse system of symbol description and creation language. The construction and storage of music data need to convey the semantic information of the original system and adapt to the characteristics of streaming media analysis and processing technology. On the other hand, in order to serve different roles, the music data widely available on the Internet usually expresses and stores the same music in different compilation forms and data formats. For example, given the same music name, the current Internet disseminates music data files in a variety of compilation methods and formats. These music data files convey similar semantics and also play different advantages: MIDI files have precise time descriptions, audio files distinguish melodies and chords by loudness and timbre, and the stave notation and fingering notation have richer decorative changes.

[0004] At present, the method of analyzing and processing heterogeneous music data is usually to predict specific category labels for the data under each arrangement method, and then perform segment alignment and label integration on the data under each arrangement method. In the integration process, the labels are statistically voted and the labels with high frequency in multiple arrangement methods are preferentially integrated as the final prediction results. The above heterogeneous parsing and fusion methods have obvious shortcomings: (1) The analysis method for music data of each arrangement method is relatively fixed and cannot utilize the potential semantic connection between the arrangement methods; (2) If the semantic connection cannot be utilized, it is impossible to correct the missing or erroneous intermediate results; (3) The analysis results of heterogeneous data are only applicable to the evaluation of one application task and cannot be reused for related music informatics analysis tasks.

[0005] To sum up, in view of the shortcomings of existing methods, it is necessary to use heterogeneous data analysis to obtain symbol sequences representing multiple music modalities, with the restoration of multiple modal music data as the learning goal, semantic interpretation and fusion of symbol sequences, and the maintenance and regulation of multiple semantic attributes as the training goal, to generate and adapt multimodal music data; and then design corresponding controllable transcription methods, customized retrieval and recommendation methods, so as to efficiently utilize heterogeneous music data and effectively solve problems such as heterogeneous data analysis, music semantic interpretation and multi-dimensional music generation quality assessment. Summary of the invention

[0006] In view of the problems existing in heterogeneous music data, such as difficulty in unifying the creative ecology and language, sparse data and independent data analysis methods, complex semantic connections and difficulty in evaluating the quality of music data, the present invention provides a heterogeneous music data fusion method and system based on symbolic semantics, which can obtain semantic attributes of different dimensions from the semantic analysis process and constrain them, realize interpretable and controllable music data fusion and generation, thereby solving the problems of heterogeneous data analysis, music semantic interpretation and multi-dimensional music generation quality assessment, and providing controllable fused data to provide services for downstream applications.

[0007] The present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a method for fusion of heterogeneous music data based on symbolic semantics, comprising the following steps:

[0009] Step 1: heterogeneous music data acquisition and feature extraction. After data preprocessing, basic information of heterogeneous music data and unique features of each data structure are obtained, and well-associated heterogeneous music data and its unified annotation sequence are obtained;

[0010] Step 2, defining a symbolic semantic parsing model, using the symbolic semantic parsing model to obtain a stable symbolic semantic observation space from the associated heterogeneous music data and its unified annotation sequence, and obtaining significant semantic attributes and controllable semantic constraints from the stable symbolic semantic observation space;

[0011] Step 3, based on the adjustment of the significant semantic attributes and controllable semantic constraints, restore or generate adapted data that conforms to specific semantic constraints.

[0012] In a second aspect, the present invention provides a heterogeneous music data fusion system based on symbolic semantics, which is used to implement the above-mentioned heterogeneous music data fusion method.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) The present invention can acquire music data in multiple data formats and multiple modes, and establish a basic framework for symbolic semantic analysis of heterogeneous data;

[0015] (2) The present invention can identify and suspend symbols with ambiguous forms and semantics in heterogeneous data, which is conducive to large-scale analysis and fusion of existing heterogeneous music data;

[0016] (3) The present invention can realize the recognition of multiple semantic attributes and the embedding of multiple semantic constraints, which is conducive to the subsequent completion and generation of corresponding data content through semantic analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of the architecture of a heterogeneous music data fusion system based on symbolic semantics according to an exemplary embodiment;

[0018] Figure 2 A schematic diagram of a method for parsing heterogeneous data according to an exemplary embodiment;

[0019] Figure 3 It is a schematic diagram of the structure of a semantic parsing model based on a VAE model according to an exemplary embodiment. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0021] The present invention is aimed at heterogeneous music data, uses symbolic semantics to perform heterogeneous analysis and fusion generation, and realizes fusion generation with the help of VAE model, providing a way to interpret and control symbolic semantics, which is conducive to alleviating the heterogeneity of music data and providing interpretable, comparable and reusable fusion information.

[0022] In a specific implementation of the present invention, a heterogeneous music data fusion method based on symbolic semantics mainly includes the following steps:

[0023] Step 1: heterogeneous music data acquisition and feature extraction. After data preprocessing, basic information of heterogeneous music data and unique features of each data structure are obtained, and well-associated heterogeneous music data and its unified annotation sequence are obtained;

[0024] Step 2, defining a symbolic semantic parsing model, using the symbolic semantic parsing model to obtain a stable symbolic semantic observation space from the associated heterogeneous music data and its unified annotation sequence, and obtaining significant semantic attributes and controllable semantic constraints from the stable symbolic semantic observation space;

[0025] Step 3, based on the adjustment of the significant semantic attributes and controllable semantic constraints, restore or generate adapted data that conforms to specific semantic constraints.

[0026] In this embodiment, a specific implementation process of step 1 is:

[0027] (1-1) Search and collect heterogeneous music data from different websites and in different data formats, including MIDI files, audio files, staff notation files, fingering notation files, and collect basic information such as music name, genre tag, composer tag, etc.

[0028] (1-2) Perform data analysis on MIDI files, extract the pitch sequences and durations of multiple tracks, and convert them into Figure 2 The rhythm interval sequence, pitch set sequence, and chroma vector sequence shown in the figure can then be used to obtain beat symbol markings, melody symbol markings, and chord symbol markings.

[0029] (1-3) Perform data analysis on the audio file, extract frequency domain features and time domain features, and use them to separate the sound source, identify the paragraphs, and estimate the pitch. Then, obtain the pitch set and its sequence and convert them into Figure 2 The paragraph distribution sequence, pitch distribution sequence and timbre distribution sequence shown in the figure are used to obtain structural symbol annotations, melody symbol annotations and chord symbol annotations.

[0030] (1-4) Data analysis is performed on the five-line notation file and the fingering notation file to extract beat identifiers, key identifiers, chord identifiers, finger position identifiers, loop identifiers, decoration identifiers, paragraph structure names, and unknown placeholders, thereby obtaining beat symbol notations, structure symbol notations, melody symbol notations, chord symbol notations, performance symbol notations, and unknown symbol notations. The specific extraction method includes:

[0031] (1-4a) For the five-line notation file, use music21 software to extract the music score layout, key signature, pitch set (combined segmented chords), decoration identifier and other information, convert the above information into key identifier, chord identifier, finger position identifier, loop identifier, decoration identifier, and then obtain beat symbol annotation, structure symbol annotation, melody symbol annotation, chord symbol annotation, and performance symbol annotation;

[0032] (1-4b) extracting the beat identifier and the key identifier from the fingering annotation file, and obtaining the beat symbol annotation according to the beat identifier;

[0033] Extract the pitch set and appearance order corresponding to the finger position of each segment to obtain the finger position identifier, and then obtain the melody symbol annotation, chord symbol annotation, and performance symbol annotation;

[0034] Extract the chord identifiers that appear in each segment, and distinguish the root note, consonant and redundant ornaments of the pitch set in the segment according to the pitch-level associations implied by the chord identifiers, obtain the ornament identifiers, and then obtain the performance symbol annotations;

[0035] If there is a melody annotation, extract the melody symbol annotation of each segment, verify the accuracy of the melody composed of the root note sequence inferred by the chord identifier, record the root note symbol annotations with differences and their positions, obtain the unknown placeholder, and then obtain the unknown symbol annotation;

[0036] If a paragraph structure name exists, record the paragraph structure name; if a loop identifier exists, mark it and copy and flatten the paragraph; obtain the structural symbol annotation according to the paragraph structure name and the loop identifier;

[0037] For the complete fingering annotation file, the pitch set and appearance order (fingering position identifier), chord identifier, digital sequence representing the melody and paragraph structure name sequence corresponding to the fingering position in each fragment are extracted. Based on the paragraph structure name, melody annotation and chord identifier, the root note that constitutes the melody, the consonants that constitute the chord, decoration identifier, loop identifier and unknown placeholder in the pitch set are distinguished.

[0038] For the simplified fingering annotation file, only the pitch set corresponding to the finger position in each segment and its appearance order and chord identifier are extracted;

[0039] (1-5) Based on the symbolic and statistical data features and corresponding time segments identified and extracted by (1-2) to (1-4), the same and unique features of the heterogeneous music data are associated through two techniques, namely, tonal equivalence calculation and structured alignment.

[0040] According to the definition of the present invention, the compilation formats of heterogeneous music data include but are not limited to MIDI files, audio files, five-line notation files, and fingering notation files. In these specific types of heterogeneous data, similar music semantics are distributed and reflected in the diverse layouts of symbols such as beats, structures, melodies, chords, and performances.

[0041] Association: Align in time and pitch, and obtain a unified annotation sequence based on the associated heterogeneous music data (see Figure 2 ).

[0042] In this embodiment, a specific implementation process of step 2 is:

[0043] 2.1 Define the encoding and learning method of label (genre label, composer label) information:

[0044] According to the prior distribution of the label T (genre label, composer label), the genre subset and composer subset in the heterogeneous music dataset are divided to obtain a uniform dataset; the timbre distribution, chord symbols, and time signatures (common in music genre analysis) are extracted from the uniformly sized genre subset and composer subset, and one-hot encoded to obtain the label semantic distribution Z; for the associated heterogeneous music data and its unified annotation sequence (heterogeneous data X), the feature encoding is obtained through the VAE encoder

[0045] Encode the label semantic distribution Z and features Splicing is used for contrastive learning in the subsequent training process; the objective function of the multi-classification label prediction learner around genres and composers is:

[0046]

[0047] Among them, f composer (.) is and f genre (.) are all multi-label classifiers, CCE(.) represents the calculation of multi-category cross entropy, h is the training parameter of contrastive learning, T is the total number of genre labels and composer labels, Representation feature encoding The i-th vector in z i represents the i-th vector in the label semantic distribution Z; h composer The training parameter representing the composer information, h genre Represents the training parameters of genre information, composer_label represents the composer label, and genre_label represents the genre label.

[0048] 2.2 Define the encoding and learning method of symbol sequence information:

[0049] The identified symbols such as beats, structures, melodies, chords, performances, and unknown placeholders in the associated heterogeneous music data and their unified annotation sequences (heterogeneous data X) are one-hot encoded. The four types of associated heterogeneous data and the unified annotation sequences are encoded using a VAE encoder. The semantic attributes encoded by the two RNN models in the VAE encoder are quantized respectively. The quantization model is a mixed Gaussian model. The quantization process adjusts μ and σ to obtain Subject to N(μ,diag(σ 2 )). μ and σ constitute the sampling space.

[0050] Six masks are used to cover the six symbols of beat, structure, melody, chord, performance and unknown placeholder. Three additional masks with lengths of 4, 8 and 16 sections are used to cover the encoding results. The melody recall rate and chord symbol recall rate are used as the dominant objective function L reconstruction Learn and restore the covered parts; the objective function L reconstruction The composition is as follows:

[0051]

[0052]

[0053] Among them, L rhythm-recall represents the melody recall rate, L chord-recall represents the chord recall rate; S represents the total number of rhythm intervals separated into 16 sections. represents the melody recall rate in the sth 16-bar rhythm interval, represents the chord recall rate in the sth 16-bar rhythm interval; similarly, 2S represents the total number of rhythm intervals separated into 8 sections. represents the melody recall rate in the s′th 8-bar rhythm interval, represents the chord recall rate in the s′th 8-bar rhythm interval, 4S represents the total number of rhythm intervals separated into 4 sections, represents the melody recall rate in the s″th 4-bar rhythm interval, It represents the chord recall rate in the s″th 4-bar rhythm interval.

[0054] 2.3 Defining semantic attributes and their verification indicators

[0055] Define a 64-dimensional latent variable space that carries semantic attributes and constraints, which is used to encode the features of samples and known features. Perform sampling learning to obtain the posterior parameter values ​​of function G and function Q for attribute observation; in the latent variable space, the ideal condition for stopping sampling learning, that is, obtaining significant semantic attributes, is that the sampling results of each latent variable can satisfy the following formula:

[0056]

[0057] At this point, a stable semantic observation space can be obtained. The objective function of learning from the latent variable space to the stable semantic observation space is:

[0058]

[0059] z attr represents the stable semantic attributes in heterogeneous music data. The condition for obtaining semantic attributes is

[0060] Among them, D(.) represents the distance calculation function between G function and Q function, G(.) represents the feature encoding The quantization result, Q(.) represents the feature encoding The reverse adjustment result of the quantization result is Representation feature encoding A semantic attribute presented in the latent variable space after sampling, represents the rth significant semantic attribute, R represents the number of significant semantic attributes, μ represents the quantization parameter in the G function, specifically the mean value between semantic attributes, σ 2 Represents the quantization parameter in the G function, specifically the variance between semantic attributes, and q represents the feature encoding The reverse adjustment parameter of , a represents the semantic attribute that is verified to be obvious, That is q attr , indicating semantic attributes Generated under the constraints of obvious semantic attribute a The learning parameters of x represent the original heterogeneous data of a piece of music and its unified annotation sequence. Represents the generated heterogeneous data and its unified annotation sequence.

[0061] 2.4 Define the objective function of high-resolution and high-fidelity restored symbol sequence:

[0062] L=α·L reconstruction +β·L regularization +γ·L context

[0063] Among them, L reconstructiaon Indicates the degree of decoupling and restoration of melody features, chord features and other features, L regularization Indicates the controllability of semantic attributes and semantic constraints, L context represents the cohesion and coupling degree of genre labels, composer labels and other information in the latent variable space, α, β, γ are weights; in this embodiment, by setting a certain number of epochs, Figure 3 The VAE encoder-decoder structure and additional labels and constraints are shown for phased iterative training.

[0064] After training the symbolic semantic parsing model using the above objective function, we obtain the significant semantic attribute a and controllable parameters μ, σ 2 ,q.

[0065] In this embodiment, a specific implementation process of step 3 is:

[0066] 3.1 The user selects semantic constraints to embed into the VAE decoder based on the stable symbolic semantic observation space and significant semantic attributes, and obtains the latent semantic attributes a′ and controllable parameters σ after density measurement c , μ c ,q attr ;

[0067] (3.1-a) The first observation basis in the stable symbolic semantic observation space comes from the vectors in the sampling space that represent the symbolic annotations of beats, structures, melody, chords, and performances. is the value on the first observation basis, Subject to N(μ c ,diag(σ c )) distribution, c k is the kth constraint in the latent variable space, μ c and σ c is the constraint distribution μ, σ in the sampling space 2 The second observation basis is the significant semantic attribute a in the latent variable space, is the value on the second observation basis; if μ of different semantic attributes c The distribution is uniform and is parameterized by Calculated is uniformly distributed on the second observation basis, that is, the distribution density on the two observation bases is uniform, then the calculated As a potential semantic attribute a′.

[0068] (3.1-b) The first observation basis and the second observation basis constitute a two-dimensional symbolic semantic observation space. By selecting different constraint distributions μ, σ of the sampling space 2 and the learning parameters q of the semantic attributes attr Observe in the two-dimensional symbolic semantic observation space, and use the distance calculation function D(.) to calculate the Euclidean distance between a and a′. When D(a,a′) = 0, it means that the potential semantic attribute a′ is the same as the original significant semantic attribute a, indicating that the semantic constraint embedding fails and the potential semantic attribute is uncontrollable; when D(a,a′) obeys a monotonic relationship and the semantic attribute a′ changes uniformly along the coordinate axis, it means that the semantic constraint embedding is successful, that is, the newly formed potential semantic attribute a′ is subject to controllable constraints, recorded as semantic constraints c, so that predictable generation results can be obtained.

[0069] In this step, the calculation process of the monotonic relationship between the original significant semantic attribute a and the newly formed potential semantic attribute a′ includes:

[0070] a. Extract the original significant semantic attribute a and potential semantic attribute a′, and set the distances between adjacent data points of the music scores or audio samples published by the same genre or composer in the sampling space and the latent variable space to be and Where M represents the total number of significant semantic attributes and potential semantic attributes, N represents the total number of latent variables, represents the i-th sampling result under the m-th semantic attribute in the latent variable space, represents the jth sampling result under the mth semantic attribute in the latent variable space, represents the weight of the nth latent variable sampling learning parameter, represents the weight of the reverse adjustment parameter of the nth latent variable, Equivalent to express represents the sampling result of all heterogeneous data with the mth semantic attribute in the latent variable space, Represents the sampling results of all heterogeneous data that do not have the mth semantic attribute in the latent variable space.

[0071] b. Given a sample with a certain semantic attribute, the sampling space distance between it and another sample belonging to the same genre or composer is D q ′, the sampling distance between samples belonging to different genres or composers is D″ q , the distance between another sample belonging to the same genre or composer in the latent variable space is D z ′, the distance between samples belonging to different genres or composers in the latent variable space is D″ z ,; if D′ q <D″ q And D z ' <D″ z , verify the existence of potential semantic attribute a′;

[0072] c. Select the existing latent semantic attribute a′ when the semantic attributes and their differences obtained from all heterogeneous data of the same genre or composer satisfy D q (a,a′)>D z (a, a′), it is proved that the sample points obey monotonic relations in the latent variable space, sampling space and sample space, and the semantic attributes are subject to controllable constraints, from which the semantic constraint c can be obtained.

[0073] (3.1-c) Perform linear interpolation on the semantic attribute a or a′ and calculate As the degree of influence of semantic constraints on the attribute observation space; represents the sum of the distances between the sampling results of the mth semantic attribute in the latent variable space, sgn(D z ) represents the sum of distances between semantic attributes in the sampling space;

[0074] Observe the influence of semantic constraints on the attribute observation space, adjust the control parameters σ, μ, q of each dimension of the sampling space and latent variable space, and record the range of changes in semantic attributes And other features until the original semantic attribute no longer exists; record the adjustable range of the control parameter q of a single semantic attribute and the weight of the affected sampling learning parameter Back propagation to obtain σ c , μ c ,q attr Other control parameters

[0075] 3.2 Latent semantic attributes a′ and controllable parameters σ after density measurement c , μ c ,q attr , controls the completion and generation of music segments.

[0076] In the completion process, in the vector quantization space, the sampled and reverse-adjusted feature encoding is performed by the original significant semantic attribute a and the potential semantic attribute a′. Decoding is performed by the VAE decoder. Get a new symbol annotation; represents a newly generated heterogeneous music clip of length s″, A={a,a′} is a set of controlled semantic attributes. The VAE decoder consists of RNN and Transformer structures. The objective function of the decoder for restoring heterogeneous data is L reconstruction .

[0077] During the generation process, the objective function of the decoder for generating heterogeneous music data is:

[0078] L improvisation =δ·L regularization +L rhythm-consistency +L chord-consistency

[0079] Where L improvisation Reflects the joint stability of the original features, L regularization Reflects the controllability of semantic attributes and semantic constraints, L rhythm-consistency Reflects the degree of deviation between the new melody and the original melody, L chord-consistency Reflects the harmony of chord progression.

[0080] Computed by the VAE decoder Get the newly generated symbol annotation, where Represents the heterogeneous music data of all newly generated music tracks.

[0081] Control parameter σ c , μ c ,q attr This controls the granularity of generating multiple semantic attributes. For example, L rhythm-consistency It is used to calculate the recall rate of the main melody pitch sequence and cyclic structure compared with the original melody, and can be used for improvisation around the semantic attributes related to chords; L chord-consistency The consistency of the pattern granularity used to define the pitch-level relations of chord symbols (such as the symbol patterns of major triad, minor triad, fifth chord, seventh chord, etc.) can be calculated, and improvisation can be generated around the semantic properties related to the chords.

[0082] Final L improvisation It is the overall objective function of controllable generation; it is trained and generated in the Transformer structure; thereby exploring the feasible space of music adaptation.

[0083] Based on the above implementation steps, users can understand heterogeneous music data according to sophisticated symbolic semantic analysis, perform multi-dimensional alignment and fusion in the latent space, and thus design corresponding controllable transcription methods, customized retrieval methods, and flexible generation and exploration methods.

[0084] In this embodiment, a heterogeneous music data fusion system based on symbolic semantics is also provided, which is used to implement the above-mentioned embodiment, and the descriptions that have been made will not be repeated. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible.

[0085] The system comprises:

[0086] Heterogeneous data acquisition module, which is used for heterogeneous music data acquisition and feature extraction, obtains basic information of heterogeneous music data and unique features of each data structure through data preprocessing, and obtains well-associated heterogeneous music data and its unified annotation sequence;

[0087] A symbolic semantic parsing module, which is used to define a symbolic semantic parsing model, and use the symbolic semantic parsing model to obtain a stable symbolic semantic observation space from the associated heterogeneous music data and its unified annotation sequence, and obtain significant semantic attributes and controllable semantic constraints from the stable symbolic semantic observation space;

[0088] The fusion generation module is used to restore or generate adapted data that conforms to specific semantic constraints based on the adjustment of significant semantic attributes and controllable semantic constraints.

[0089] In addition, an interaction module can be designed and configured in the system. The interaction module is responsible for the interaction between the entire system and the user, so that the user can parse heterogeneous data and observe and control semantic attributes from the perspective of multiple attribute densities.

[0090] An optional working process of the above system is as follows:

[0091] (1) The heterogeneous data acquisition module acquires and extracts features of heterogeneous music data. After data preprocessing, the basic information of the heterogeneous music data and the unique features of each data structure are obtained, and the associated heterogeneous music data and its unified annotation sequence are obtained.

[0092] (1.1) Collect heterogeneous music data such as images, audio and MIDI files, align the multi-granularity time segments of the heterogeneous data, and extract multiple perceptual features and chord symbol sequences; the process and objects of data analysis are as follows: Figure 1 shown.

[0093] (1.2) Paragraph alignment and equivalence calculation. The specific steps are as follows:

[0094] Firstly, the main melody sequence extracted from the bass track of the MIDI file and the melody sequence of the fingering notation file are associated with the structure segment name sequence;

[0095] Secondly, the main melody sequence extracted from the bass track of the MIDI file and the chromaticity vector sequence composed of all pitch sets are associated with the pitch sequence in the audio file, and the various instrumentation tracks of the MIDI file are associated with the timbre distribution in the audio, thereby distinguishing melodies, chords, and other ornaments; the Dynamic Time Warping (DTW) algorithm is used to calculate the similarity matrix of the chromaticity maps from the audio file and the MIDI file, decompose the distance matrix and the path matrix, and determine the most similar audio and MIDI segments by the shortest path. Notes and segments with similarity below a certain threshold are marked;

[0096] Thirdly, the melody sequence of the fingering annotation file is associated with the pitch sequence extracted from the audio file, the chord identifier in the fingering annotation is associated with the estimated chord symbol in the audio file, and the loop ornament in the fingering annotation is associated with the loop segment annotation in the audio file; the unknown identifier in the fingering annotation and the chord symbol that cannot be matched in the audio file are marked; the equivalence calculation formula for the key is:

[0097] rot(x,n,i)=(x<<i)|(x>>(ni))&(2 n -1)

[0098] reduce(x,n)=min({rot(x,n,i):0≤i <n})

[0099] pcd(x)=reduce(x,12)

[0100] Through the rot algorithm's tonal equivalence calculation, transposed scores or audio with lower sound quality can obtain more relevant parsing results. In the chord identification structure of General Chord Type, the calculation result of pcd(x) can merge the fine chord levels into large categories of chord identifiers, simplify the representation and learning of chord symbols, and reduce the semantic parsing accuracy without losing the original harmony.

[0101] Finally, the identifiers of heterogeneous music data are eliminated and merged according to the known key signature range in the data and the semitone relationship calculated by the tonal equivalence calculation formula to enhance the existing associations; forming melody sequences and chord sequences with unknown placeholders and multiple candidate note sets.

[0102] (2) The symbolic semantic parsing module defines a symbolic semantic parsing model, uses the symbolic semantic parsing model to obtain a stable symbolic semantic observation space from the associated heterogeneous music data and its unified annotation sequence, and obtains significant semantic attributes and controllable semantic constraints from the stable symbolic semantic observation space;

[0103] (2.1) RNN and comparable masks are used to sample and learn various symbol sequences; functions such as μ, σ, q are implemented by the RNN structure, and functions G and Q are implemented by the Transformer structure.

[0104] (2.2) Define the semantic attributes to be observed, such as a fixed chord progression sequence within 8 bars, and the chord granularity of the same level in the chord progression sequence (such as the granularity of the fifth chord, the seventh chord, etc., which is expressed as a different set of pitches);

[0105] (2.3) Define the loss function for reconstruction and restoration and conduct training;

[0106] (2.4) Based on the distribution of the semantic attribute space and the semantic attributes that have been verified to be significant, the user selects semantic constraints to be embedded in the loss function of the fine-tuning stage or the new training stage, thereby controlling the extent to which some semantic attributes play a role in the fusion generation process.

[0107] (3) The fusion generation module restores or generates adapted data that conforms to specific semantic constraints based on the adjustment of salient semantic attributes and controllable semantic constraints.

[0108] (3.1) By observing the density measurement results of semantic attributes and semantic constraints, the influence range of semantic attributes and semantic constraints is observed and verified, and the completion of the music segment is controlled.

[0109] (3.2) By observing the density measurement results of semantic attributes and semantic constraints, the influence range of semantic attributes and semantic constraints is observed and verified, and the generation of music segments is controlled.

[0110] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fusion of heterogeneous music data based on symbolic semantics, characterized in that: The following steps are involved: Step 1: heterogeneous music data acquisition and feature extraction. After data preprocessing, basic information of heterogeneous music data and unique features of each data structure are obtained, and well-associated heterogeneous music data and its unified annotation sequence are obtained; The step 1 comprises: (1-1) Search and collect heterogeneous music data from different websites and in different data formats, including MIDI files, audio files, five-line notation files, fingering notation files, and collect basic information such as music name, genre tag, and composer tag; (1-2) Analyze the MIDI file data, extract the pitch sequences and durations of multiple tracks, convert them into rhythm interval sequences, pitch set sequences, and chromaticity vector sequences, and then obtain beat symbol notations, melody symbol notations, and chord symbol notations; (1-3) Performing data analysis on the audio file, extracting frequency domain features and time domain features, and using them to perform sound source separation, paragraph recognition, and pitch estimation, obtaining pitch sets and their sequences, and converting them into paragraph distribution sequences, pitch distribution sequences, and timbre distribution sequences, and then obtaining structural symbol annotations, melody symbol annotations, and chord symbol annotations; (1-4) performing data analysis on the five-line notation file and the fingering notation file, extracting beat identifiers, key identifiers, chord identifiers, finger position identifiers, loop identifiers, decoration identifiers, paragraph structure names, and unknown placeholders, and then obtaining beat symbol notations, structure symbol notations, melody symbol notations, chord symbol notations, performance symbol notations, and unknown symbol notations; (1-5) for the symbolic and statistical data features and corresponding time segments identified and extracted in steps (1-2) to (1-4), the same and unique features of the heterogeneous music data are associated through tonality equivalence calculation and structured alignment technology to obtain associated heterogeneous music data and a unified annotation sequence; Step 2, defining a symbolic semantic parsing model, using the symbolic semantic parsing model to obtain a stable symbolic semantic observation space from the associated heterogeneous music data and its unified annotation sequence, and obtaining significant semantic attributes and controllable semantic constraints from the stable symbolic semantic observation space; The definition of the symbolic semantic parsing model includes defining the encoding and learning methods of the tag information and the symbol sequence, the semantic attributes and the verification index, and the objective function of restoring the symbol sequence; after training the symbolic semantic parsing model using the objective function, the significant semantic attribute a and the controllable parameters μ, σ are obtained. 2 ,q; The tag information includes genre tags and composer tags; the encoding and learning methods for defining the tag information include: According to the prior distribution of labels, the genre subsets and composer subsets in the heterogeneous music dataset are divided to obtain uniform datasets; the timbre distribution, chord symbol and time signature features are extracted from the uniform genre subsets and composer subsets, and one-hot encoded to obtain the label semantic distribution Z; the associated heterogeneous music data and its unified annotation sequence are recorded as heterogeneous data X, and the feature encoding is obtained through the VAE encoder. Encode the label semantic distribution Z and features Splicing is used for contrastive learning in the subsequent training process, and the objective function is: Among them, f composer (.) is and f genre (.) are all multi-label classifiers, CCE(.) represents the calculation of multi-category cross entropy, h is the training parameter of contrastive learning, T is the total number of genre labels and composer labels, Representation feature encoding The i-th vector in z i represents the i-th vector in the label semantic distribution Z; h composer The training parameter representing the composer information, h genre Represents the training parameters of genre information, composer_label represents the composer label, and genre_label represents the genre label; The encoding and learning methods for defining symbol sequences include: The associated heterogeneous music data and their unified annotation sequence are recorded as heterogeneous data X. The identified beats, structures, melodies, chords, performances, and unknown placeholders in the heterogeneous data X are one-hot encoded. The four types of associated heterogeneous data and the unified annotation sequence are encoded using a VAE encoder. The semantic attributes encoded by the two RNN models in the VAE encoder are then quantized. Six masks are used to cover the six symbols of beat, structure, melody, chord, performance and unknown placeholder. Three additional masks with lengths of 4, 8 and 16 sections are used to cover the encoding results. The melody recall rate and chord symbol recall rate are used as the dominant objective function L reconstruction Learn and restore the covered parts; the objective function L reconstruction The composition is as follows: Among them, L rhythm-recall represents the melody recall rate, L chord-recall represents the chord recall rate; S represents the total number of rhythm intervals separated into 16 sections. represents the melody recall rate in the sth 16-bar rhythm interval, represents the chord recall rate in the sth 16-bar rhythm interval; similarly, 2S represents the total number of rhythm intervals separated into 8 sections. represents the melody recall rate in the s′th 8-bar rhythm interval, represents the chord recall rate in the s′th 8-bar rhythm interval, 4S represents the total number of rhythm intervals separated into 4 sections, represents the melody recall rate in the s″th 4-bar rhythm interval, represents the chord recall rate in the s″th 4-bar rhythm interval; Definition of semantic attributes and verification indicators include: Define a 64-dimensional latent variable space that carries semantic attributes and constraints, which is used to encode the features of samples and known features. Perform sampling learning to obtain the posterior parameter values ​​of function G and function Q for attribute observation; in the latent variable space, the ideal condition for stopping sampling learning, that is, obtaining significant semantic attributes, is that the sampling results of each latent variable can satisfy the following formula: At this point, a stable semantic observation space can be obtained. The objective function of learning from the latent variable space to the stable semantic observation space is: Among them, z attr represents the stable semantic attributes in heterogeneous music data; D(.) represents the distance calculation function between G function and Q function, and G(.) represents the feature encoding The quantization result, Q(.) represents the feature encoding The reverse adjustment result of the quantization result is Representation feature encoding A semantic attribute presented in the latent variable space after sampling, represents the rth significant semantic attribute, R represents the number of significant semantic attributes; μ represents the quantization parameter in the G function, specifically the mean value between semantic attributes; σ 2 Represents the quantization parameter in the G function, specifically the variance between semantic attributes; q represents the feature encoding The reverse adjustment parameter of , a represents the semantic attribute that is verified to be obvious; That is q attr , indicating semantic attributes Generated under the constraints of obvious semantic attribute a The learning parameters of x are: Represent the generated heterogeneous data and its unified annotation sequence; The objective function of restoring the symbol sequence is: L=α·L reconstruction +β·L regularization +γ·L context Among them, L reconstruction Indicates the degree of decoupling and restoration of melody features, chord features and other features, L regularization Indicates the controllability of semantic attributes and semantic constraints, L context It represents the cohesion and coupling degree of genre labels, composer labels and other information in the latent variable space, and α, β, and γ are weights; Step 3, based on the adjustment of the significant semantic attributes and the controllable semantic constraints, restore or generate adapted data that conforms to the specific semantic constraints; The step 3 comprises: (3-1) The user selects semantic constraints to embed into the VAE decoder based on the stable symbolic semantic observation space and significant semantic attributes, and obtains the latent semantic attributes a′ and controllable parameters σ after density measurement c , μ c ,q attr ; (3-2) The potential semantic attribute a′ and controllable parameter σ after density measurement c , μ c ,q attr , control the completion and generation of music segments: In the completion process, the sampled and reverse-adjusted feature encoding is performed by the original significant semantic attribute a and the potential semantic attribute a′. Decoding is performed, which is calculated by the VAE decoder Get a new symbol annotation; represents a newly generated heterogeneous music segment of length s″, A={a,a′} is the set of controlled semantic attributes; In the generation process described above, the VAE decoder calculates Get the newly generated symbol annotation, Representing heterogeneous music data of all newly generated music tracks; The step (3-1) comprises: (3.1-a) The first observation basis in the stable symbolic semantic observation space comes from the vectors in the sampling space that represent the beat, structure, melody, chord and performance symbol annotations, is the value on the first observation basis, Subject to N(μ c ,diag(σ c )) distribution, c k is the kth constraint in the latent variable space, μ c and σ c is the constraint distribution μ, σ in the sampling space 2 Based on the sampling learning and feedback propagation of the control parameters; the second observation basis is the significant semantic attribute a in the latent variable space, is the value on the second observation basis; if μ of different semantic attributes c The distribution is uniform and is parameterized by Calculated Evenly distributed on the second observation basis, that is, the distribution density on the two observation bases is uniform, then the calculated as a potential semantic attribute a′; (3.1-b) The first observation basis and the second observation basis constitute a two-dimensional symbolic semantic observation space. By selecting different constraint distributions μ, σ of the sampling space 2 and the learning parameters q of the semantic attributes attr Observe in the two-dimensional symbolic semantic observation space and use the distance calculation function D(.) to calculate the Euclidean distance between a and a′; when D(a, a ′ )=0, it means that the potential semantic attribute a′ is the same as the original significant semantic attribute a, indicating that the semantic constraint embedding fails and the potential semantic attribute is uncontrollable. ′ ) obeys a monotonic relationship, and the semantic attribute a ′ As the coordinate axis changes evenly, it means that the semantic constraint is embedded successfully, that is, the newly formed potential semantic attribute a′ is subject to controllable constraints, recorded as semantic constraint c, so that predictable generation results can be obtained; (3.1-c) Perform linear interpolation on the semantic attribute a or a′ and calculate As the degree of influence of semantic constraints on the attribute observation space; represents the sum of the distances between the sampling results of the mth semantic attribute in the latent variable space, sgn(D z ) represents the sum of distances between semantic attributes in the sampling space; Observe the influence of semantic constraints on the attribute observation space, adjust the control parameters σ, μ, q of each dimension of the sampling space and latent variable space, and record the range of changes in semantic attributes And other features until the original semantic attribute no longer exists; record the adjustable range of the control parameter q of a single semantic attribute and the weight of the affected sampling learning parameter Back propagation to obtain σ c , μ c ,q attr Control parameters.

2. A heterogeneous music data fusion system based on symbolic semantics, used to implement the heterogeneous music data fusion method described in claim 1, characterized in that: The system comprises: Heterogeneous data acquisition module, which is used for heterogeneous music data acquisition and feature extraction, obtains basic information of heterogeneous music data and unique features of each data structure through data preprocessing, and obtains well-associated heterogeneous music data and its unified annotation sequence; A symbolic semantic parsing module, which is used to define a symbolic semantic parsing model, and use the symbolic semantic parsing model to obtain a stable symbolic semantic observation space from the associated heterogeneous music data and its unified annotation sequence, and obtain significant semantic attributes and controllable semantic constraints from the stable symbolic semantic observation space; The fusion generation module is used to restore or generate adapted data that conforms to specific semantic constraints based on the adjustment of significant semantic attributes and controllable semantic constraints.

Images