Symbolized polyphonic data anomaly detection and error correction method based on heterogeneous graph neural network and related equipment

Abstract

The invention discloses a symbolized polyphonic data anomaly detection and error correction method based on a heterogeneous graph neural network and related equipment. The method comprises the following steps: analyzing a symbolized polyphonic data stream, and extracting a multi-dimensional acoustic feature vector; a heterogeneous topological graph G = (V, Et, Ec) is constructed, Et is a horizontal directed edge, and Ec is a vertical undirected edge; extracting and fusing space-time features through a space-time diagram convolutional network; calculating a matching degree weight w based on the statistical constraint matrix, and positioning a topological distortion node cluster through threshold judgment and a connected component algorithm; and generating a candidate node set for the distorted node cluster, optimizing and correcting errors by using a global cost function, and outputting a reconstructed sequence. According to the method, the structural sensitivity and robustness of the abnormal positioning of the polyphonic data are improved, the error correction efficiency is improved, and the method can be widely applied to scenes such as music education, creation assistance and music score editing.

Description

Technical Field

[0001] This invention relates to the fields of computer music information processing, graph neural networks and data cleaning and error correction technology, specifically to a method for anomaly detection and error correction of symbolic polyphonic data based on heterogeneous graph neural networks, and also to a system, electronic device and computer-readable storage medium for implementing the method. Background Technology

[0002] Symbolic polyphonic music data is widely used in music education, creative assistance, music retrieval, automatic accompaniment and score editing, etc. It is usually represented as a digital polyphonic music event stream, which can be parsed by formats such as MIDI and MusicXML. In the process of automatic transcription, batch format conversion, manual editing and multi-source fusion, the above event stream is prone to structural anomalies, including: (1) Simultaneous part timing anomalies: inconsistent continuous pitch or octave jumps, or unsmooth transitions between adjacent event states; (2) Cross-part concurrency anomalies: low probability and incompatible interval combinations appear in cross-part concurrency relationships within the same time window; (3) Chain errors caused by local anomalies: an anomaly in one part will affect the concurrency relationship and subsequent timing transitions, resulting in propagation deviations.

[0003] While various error detection / correction approaches exist in existing technologies, they still fall short in terms of structured anomaly localization and global consistency error correction for polyphonic data. For example: 1) The "Audio-Score Comparison Method with Error Detection Function" disclosed in patent application publication number CN103354092A mainly achieves audio-score alignment and detects deviations through chroma vectors and dynamic time warping. It focuses on audio-score alignment and deviation detection, but has limited support for joint modeling of cross-voice concurrent topology and same-voice temporal topology in polyphonic event streams, making it difficult to achieve decoupled expression and global consistency error correction based on heterogeneous topology; 2) The "Harmony Learning Method, Device and Electronic Equipment" disclosed in patent application publication number CN111968452A focuses on judging and correcting errors in four-part harmonic chords. The false prompts belong to the rule / judgment-based analysis path, which has limited support for robustness and generalizable statistical consistency constraint expression in multi-style and multi-source noise scenarios, and is difficult to achieve probabilistic matching and localization and global cost optimization reconstruction based on statistical topological constraint matrix; 3) The "A Method, Device, Equipment and Storage Medium for Generating Ethnic Music" disclosed in the authorization announcement number CN113035162B uses LSTM and other sequence modeling units to generate materials, which is insufficient in expressing cross-voice concurrent constraints in polyphonic data, and is difficult to simultaneously perform heterogeneous topological modeling of concurrency and temporal dependence and achieve distortion cluster-level error correction.

[0004] Furthermore, in the processing of high-density polyphonic data, existing models lack a decoupling mechanism for "cross-voice spatial constraints" and "single-voice temporal constraints," leading to feature collapse or structural drift during data reconstruction. This makes it difficult to maintain global topological consistency while ensuring local corrections. Therefore, there is an urgent need for a technical solution that can construct polyphonic events as heterogeneous topological structures containing both intra-voice temporal dependencies and cross-voice concurrent dependencies, extract spatiotemporal features through message passing and weighted calculation mechanisms of different edge types, and then complete anomaly localization and global consistency error correction under statistical constraints. Summary of the Invention

[0005] Purpose of the invention The purpose of this invention is to provide a method and related equipment for anomaly detection and correction of symbolic polyphonic data based on heterogeneous graph neural networks, which realizes anomaly localization of digital polyphonic music event streams and outputs the reconstructed event stream after error correction, improves the structural sensitivity and robustness of anomaly localization, reduces the candidate search space and improves the global consistency of error correction results. Technical solution

[0006] To achieve the above objectives, the present invention employs the following technical solution: parsing the input symbolic polyphonic data stream, extracting a multi-dimensional acoustic feature vector containing pitch class index, octave cardinality, start timestamp, duration value, and voice part index, and mapping each note event to an event node in the graph; mapping the multi-dimensional feature vector to a heterogeneous topological graph G=(V,Et,Ec), where the horizontal directed edge Et represents the temporal state transition relationship within the same logical voice part, and the vertical undirected edge Ec represents the cross-voice relationship within a preset time window length Δt. Concurrent interference relationships are used to simultaneously express the temporal dependence of the same voice and the concurrent dependence of the cross voice in polyphonic data; Δt is dynamically calculated from global beat or velocity information. Assuming the unit beat length is the duration T, Δt is constrained within the absolute millisecond time span corresponding to 0.5T to 1.0T. When the data stream lacks global beat or velocity information, Δt takes a preset fixed value. The heterogeneous topology graph is input into the trained spatiotemporal graph convolutional network, extracting temporal evolution features along Et and concurrent structural features along Ec through spatial graph attention. The spatiotemporal fusion feature vector of the node is obtained by fusing two types of features. The fusion method includes at least one of tensor concatenation, gated fusion unit, and self-attention weighted fusion. A preset statistical topological constraint matrix is ​​called, which includes a concurrency compatibility constraint matrix and a sequence consistency constraint matrix. The concurrency compatibility probability pc is extracted for concurrent edges, and the temporal consistency probability pt is extracted for temporal edges. The weighted sum is obtained by weighting according to the preset feature weight coefficients α and β to obtain s = α·pc + β·pt. The matching degree weight w is obtained by the activation function σ. When w is lower than the judgment threshold θ, abnormal edges and their connected nodes are extracted to construct an induced subgraph. After the induced subgraph is undirected, the connected components are extracted using the undirected graph connected component algorithm, and the connected components are marked as topological distortion node clusters. Multiple candidate node combinations are generated for the topological distortion node clusters. The minimum cost path is selected under the global cost function constraint by using a sequence decoding and optimization algorithm. Under the condition that the start timestamp, duration value and voice part index remain unchanged, the pitch class index and the octave cardinality parallel connection relationship are replaced, and the error-corrected symbolic polyphonic sequence is output. Beneficial effects

[0007] Compared with the prior art, the present invention has the following beneficial effects: (1) It achieves decoupling expression of temporal dependence and concurrent dependence through the heterogeneous topology of Et and Ec, thereby improving the structural sensitivity of anomaly localization; (2) It obtains the topological constraint matrix based on corpus statistics, transforms compatibility and consistency into computable probability indicators and forms a threshold judgment mechanism for matching degree weight w, thereby improving the robustness across data source scenarios; (3) It obtains a cluster of distorted nodes through connected component clustering, and then performs joint optimization through global cost function, taking into account both error correction efficiency and global consistency; (4) It introduces the constraints of voice range threshold and semitone span K in the candidate generation stage, significantly reducing the size of the candidate set, improving optimization efficiency and reducing computational overhead. Attached Figure Description

[0008] The embodiments of the present invention will be further described below with reference to the accompanying drawings. The drawings are for illustrative purposes only and do not constitute a limitation on the scope of protection of the present invention. Figure 1 A schematic diagram of the overall process of the symbolic polyphonic data anomaly detection and correction method based on heterogeneous graph neural network provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the heterogeneous topology graph G=(V,Et,Ec) provided in an embodiment of the present invention, showing the connection relationship between the horizontal directed edge Et and the vertical undirected edge Ec. Figure 3 This is a schematic diagram of the spatiotemporal graph convolutional network structure provided in an embodiment of the present invention, illustrating the message passing and feature extraction process between the temporal convolutional kernel and the spatial graph attention layer; Figure 4 This is a schematic diagram illustrating the principle of topological distortion localization and connected component clustering provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the module structure of the anomaly detection and error correction system provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0009] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The scope of protection of the present invention is not limited to the following embodiments. Equivalent substitutions or modifications made by those skilled in the art without departing from the spirit of the present invention should be included in the scope of protection of the present invention.

[0010] Example 1: Event Flow Parsing and Node Feature Construction The input is a digitized polyphonic music event stream. For each note event i, at least the following fields are extracted: pitch index pc_i, octave cardinality oct_i, start timestamp t_i, duration d_i, and voice part index v_i. The node feature vector x_i=[pc_i,oct_i,t_i,d_i,v_i] is constructed. Without affecting the core process of this invention, additional dimensions such as velocity and instrument channel can also be added to the feature vector.

[0011] Example 2: Construction of Heterogeneous Topology Graph (1) Node set V: Map each note event to a node v_i, and the node set is V={v_1,…,v_N}; (2) Horizontal directed edge Et: Sort the node sets with the same index for each voice part according to the start time, and add a directed edge (v_i→v_j) to the adjacent successor node to form Et; (3) Vertical undirected edge Ec: Set the time window length Δt, and add an undirected edge {v_i,v_j} to the cross-voice node pair (v_i,v_j) when |t_i−t_j| does not exceed Δt to form Ec; (4) Determination of Δt: If the data stream contains BPM or beat information, then the duration of each beat T=60 / BPM (seconds) and convert it to milliseconds, and Δt takes the millisecond span corresponding to 0.5T to 1.0T; if the speed information is missing, then Δt takes a preset fixed value.

[0012] Example 3: Training and Inference of Spatiotemporal Graph Convolutional Networks A heterogeneous topological graph is input into a spatiotemporal graph convolutional network, and message passing with different parameters is performed along Et and Ec respectively: temporal evolution features are extracted along Et using temporal convolutional kernels, and concurrent structural features are extracted along Ec using spatial graph attention layers. The spatiotemporal fusion feature vector h_i of the nodes is obtained through tensor concatenation, gated fusion units, or self-attention weighted fusion. During the training phase, standard sequences in the corpus are used as positive samples, and random perturbations are injected into the standard sequences to generate distorted sequences as negative samples. Perturbation methods include randomly replacing the pitch class index and / or octave cardinality of some nodes, destroying local concurrent relationships, and introducing low-probability state transitions. The network is trained using cross-entropy loss and / or contrastive loss so that the network learns to identify inconsistencies in the topological structure. In an exemplary embodiment, the spatiotemporal graph convolutional network adopts a 2-layer stacked structure, the spatial graph attention layer has 4 attention heads, and the temporal convolutional kernel size is 3.

[0013] Example 4: Construction of Statistical Topological Constraint Matrix (1) Concurrency compatibility constraint matrix: Traverse the corpus, count the co-occurrence count of interval category index of concurrent node pairs across voice parts within the time window Δt, and normalize the count to obtain the concurrency compatibility probability table; (2) Sequence consistency constraint matrix: Traverse the corpus, count the co-occurrence count of state transition category index of adjacent temporal edges within each voice part and normalize it to obtain the temporal consistency probability table; (3) The corpus adopts publicly available standard symbolized polyphonic datasets, such as the MAESTRO dataset and the POP909 dataset.

[0014] Example 5: Topological Distortion Localization and Connected Component Clustering (1) Matching weight calculation: Obtain the concurrent compatibility probability pc for concurrent edges in Ec, and the temporal consistency probability pt for temporal edges in Et, and calculate it according to the following formula: s=α·pc+β·pt; w=σ(s), where α and β are preset feature weight coefficients, σ ​​is the Sigmoid activation function, and w takes the value in the (0,1) interval; (2) Judgment threshold θ: When w is less than the judgment threshold θ, the corresponding edge is marked as an abnormal edge; θ is set according to the statistical distribution of w in the corpus and the fault tolerance requirement. In an exemplary embodiment, θ takes the value of 0.3 to 0.6; (3) Connected component extraction: Extract the abnormal edge and its connected nodes to form an induced subgraph. After the directed edge Et in the induced subgraph is undirected, it is combined with Ec to form an undirected induced subgraph. The DFS or BFS algorithm is used to calculate the connected component set. Each connected component is a cluster of topological distortion nodes.

[0015] Example 6: Candidate Generation and Global Optimization and Error Correction (1) Candidate generation: Keep the node start timestamp, duration, and voice index unchanged, and only replace the pitch class index and octave cardinality; define the absolute pitch value P = 12·octave cardinality + pitch class index, and under the upper and lower limit threshold constraints of the voice range, filter the candidate pitch class index and corresponding octave cardinality that satisfy |P_cand−P_orig|≤K to generate candidate nodes, where the value of K ranges from 1 to 5 semitones; (2) Cost function: The cost function satisfies the following form: Cost = λ1·Σ_{(va,vb)∈Ec} D(va,vb) + λ2·Σ_{(vt,vt+1)∈Et} M(vt,vt+1) + λ3·ΔF; where D(va,vb)=-log(pc(va,vb)+ε), M(vt,vt+1)=-log(pt(vt,vt+1)+ε), ε is a preset positive smoothing constant and 0<ε≤10^{-6}; ΔF is the Frobenius norm of the difference between the candidate node combination and the original topological distorted node cluster feature matrix; λ1, λ2, λ3 are preset positive weight hyperparameters; (3) Optimization algorithm: at least one of the Viterbi algorithm and the beam search algorithm is used as the sequence decoding and optimization algorithm, and the path with the minimum global state transition cost is selected to complete the error correction.

[0016] Example 7: System Example This invention also provides a symbolic polyphonic data anomaly detection and correction system, comprising: a data parsing module: used to acquire the input symbolic polyphonic data stream, wherein the symbolic polyphonic data stream is a digital polyphonic music event stream, and to parse and extract multi-dimensional acoustic feature vectors from the data stream, wherein the multi-dimensional acoustic feature vectors include at least pitch class index, octave cardinality, start timestamp, duration value, and voice part index; a heterogeneous graph construction module: used to execute the step of constructing a heterogeneous topology graph in step S1, and to construct a heterogeneous topology graph containing a double edge structure of Et and Ec; a spatiotemporal feature extraction and anomaly localization module: used to execute the step of spatiotemporal dual-dimensional feature extraction and fusion in step S2 and the step of topology distortion localization in step S3, and to output the connected components marked as topology distortion node clusters; and a state optimization and reconstruction module: used to execute the step of state space optimization and error correction output in step S4, and to optimize and output the reconstructed symbolic polyphonic sequence based on the global state transition cost.

[0017] Example 8: Electronic Devices and Storage Media The present invention also provides an electronic device, including at least one processor and a memory, wherein the memory stores instructions executable by the processor, and the processor executes the instructions to implement the symbolic polyphonic data anomaly detection and error correction method of the present invention; the present invention also provides a computer-readable storage medium storing computer instructions, wherein when the computer instructions are executed, the computer implements the symbolic polyphonic data anomaly detection and error correction method of the present invention.

[0018] Effect verification (exemplary test results) To verify the technical effect of the present invention, a comparative experiment was conducted on the publicly available standard symbolic polyphonic dataset test set, and the following exemplary results were obtained: (1) Anomaly detection accuracy: The present invention achieved 94.2%, which is 12.5% ​​higher than the LSTM-based sequence model and 18.7% higher than the traditional rule matching method; (2) Error correction accuracy: The present invention achieved 91.7%, which is 15.3% higher than the local heuristic repair method; (3) Processing efficiency: The average processing time for a single 4-bar 4-voice sequence is 12ms, which is 68% shorter than the global optimization scheme without search space constraints. The above experimental data are based on the exemplary test scenario, and the scope of protection of the present invention is not limited by the above values. Example

[0019] The difference between this embodiment and the previous embodiment is that the horizontal directed edge Et also includes non-adjacent k-order temporal edges within the same voice part, which are used to capture long-distance temporal dependencies in polyphonic data and improve the anomaly detection accuracy of long sequences; the value of k can be adjusted according to the sequence length, and this invention does not limit it. Example

[0020] The difference between this embodiment and the previous embodiments is that, in addition to MIDI and MusicXML formats, the symbolic polyphonic data stream also includes guitar tablature, piano roll format, and digital note event streams obtained by ABC notation parsing. The technical solution of this invention is applicable to data streams of the above formats.

Claims

1. A method for anomaly detection and error correction in symbolic polyphonic data based on heterogeneous graph neural networks, the method comprising: The input symbolic polyphonic data stream is obtained, which is a digital polyphonic music event stream. The multi-dimensional acoustic feature vector in the data stream is parsed and extracted. The multi-dimensional acoustic feature vector includes at least pitch class index, octave cardinality, start timestamp, duration value and voice part index. The method is characterized by further comprising the following steps: Step S1: Construct a heterogeneous topology graph: Map the multi-dimensional acoustic feature vectors to a heterogeneous topology graph G=(V, Et,Ec), where V is the set of discrete note event nodes; Et is the set of horizontal directed edges representing the sequential temporal state transitions of nodes within the same logical voice based on voice index matching; Ec is the set of vertical undirected edges representing the concurrent interference relationship of cross-voice nodes within a preset time window length Δt, where the difference in starting timestamps does not exceed Δt; wherein, Δt is dynamically calculated from the global beat or velocity information in the symbolic polyphonic data stream, and the unit beat length is T, and Δt is constrained within the absolute time span (milliseconds) corresponding to [0.5T, 1.0T]; when the symbolic polyphonic data stream has no global beat or velocity information, Δt takes a preset fixed value; Step S2: Spatiotemporal Dual-Dimensional Feature Extraction and Fusion: The heterogeneous topological graph G is input into a trained spatiotemporal graph convolutional network, which includes a temporal convolutional kernel and a spatial graph attention layer. Different parameters are used for message passing and attention weight calculation on the horizontal directed edge Et and the vertical undirected edge Ec, respectively. Temporal evolution features are extracted along the horizontal directed edge Et using the temporal convolutional kernel, and spatial structure features of concurrent nodes are calculated along the vertical undirected edge Ec using the spatial graph attention layer. The temporal evolution features and spatial structure features are fused to obtain the spatiotemporal fusion feature vector of each node. Step S3: Topological Distortion Localization Based on Statistical Constraint Matrix: A preset statistical topological constraint matrix is ​​invoked, which includes a concurrency compatibility constraint matrix and a sequence consistency constraint matrix. For each vertical undirected edge e∈Ec, an interval category index is derived based on the pitch category index of the node pairs it connects to, and the corresponding concurrency compatibility probability pc is extracted from the concurrency compatibility constraint matrix. For each horizontal directed edge e∈Et, a state transition category index is derived based on the pitch category index and octave cardinality of the nodes it connects to, and the corresponding temporal consistency probability pt is extracted from the sequence consistency constraint matrix. pc and pt are weighted and summed using preset feature weight coefficients α and β to obtain s = α·pc + β·pt, and s is input into the activation function σ to obtain the matching degree weight w of the corresponding local topology. When the matching degree weight w of the local topology is lower than a preset judgment threshold, the corresponding abnormal edge and its connected nodes are extracted, and connected components are extracted on the induced subgraph of the heterogeneous topology graph G using the undirected graph connected component algorithm. The extracted connected components are marked as a topological distortion node cluster. Step S4: State Space Optimization and Error Correction Output: For the topologically distorted node cluster, the sequence decoding and optimization algorithm is called to generate multiple candidate node combinations; the global state transition cost of each candidate combination under the statistical topological constraint matrix is ​​calculated using the cost function, and the path that minimizes the global state transition cost is selected. Under the condition that the start timestamp, duration, and voice part index of the original node input remain unchanged, the pitch class index and octave cardinality replacement and edge relationship reconnection are performed on the nodes in the topologically distorted node cluster to generate and output the reconstructed symbolic polyphonic sequence.

2. The method according to claim 1, characterized in that, The activation function σ in step S3 is the Sigmoid activation function that maps real numbers to the interval (0,1).

3. The method according to claim 1, characterized in that, The feature fusion method in step S2 includes at least one of tensor splicing, gated fusion unit, and self-attention weighted fusion.

4. The method according to claim 1, characterized in that, The sequence decoding and optimization algorithm in step S4 includes at least one of the Viterbi algorithm and the beam search algorithm.

5. The method according to claim 1, characterized in that, The global state transition cost satisfies the formula: Cost = λ1·Σ_{(va,vb)∈Ec} D(va,vb) + λ2·Σ_{(vt,vt+1)∈Et} M(vt,vt+1)+ λ3·ΔF; Where D(va,vb) represents the compatibility penalty cost on concurrent edges, defined as the negative log probability with a smoothing term -log(pc(va,vb)+ε); M(vt,vt+1) represents the consistency transition cost on temporal edges, defined as the negative log probability with a smoothing term -log(pt(vt,vt+1)+ε); where ε is a preset positive smoothing constant to prevent meaningless logarithmic calculations, and 0<ε≤10^{-6}; ΔF represents the calculation deviation between the candidate node combination and the feature matrix of the original topological distorted node cluster, defined as the Frobenius norm of the matrix difference ‖X_cand−X_orig‖_F; λ1, λ2, and λ3 are preset positive weight hyperparameters.

6. The method according to claim 1, characterized in that, The specific steps in step S4 of calling the sequence decoding and optimization algorithm to generate multiple candidate node combinations include: extracting the upper and lower limit thresholds of the vocal range of each original node in the topological distortion node cluster, wherein the threshold is a preset threshold or obtained statistically from a standard symbolic polyphonic data corpus; generating a candidate node set within the constraint range of the upper and lower limit thresholds of the vocal range to limit the optimization search space of the cost function; wherein the absolute pitch value of the node is defined as P = 12·octave cardinality + pitch class index, and candidate pitch class indices and corresponding octave cardinality that satisfy |P_cand−P_orig|≤K are selected to generate candidate nodes, wherein the preset maximum span K ranges from 1 to 5 semitones.

7. The method according to claim 1, characterized in that, The statistical topological constraint matrix is ​​obtained through the following steps: acquiring standard symbolized polyphonic data as a corpus; statistically counting the co-occurrence of target feature dimensions within the preset time window length Δt in the corpus to generate a feature count matrix; The feature counting matrix is ​​frequency normalized, and the joint probability distribution is calculated to generate a concurrency compatibility constraint matrix representing the concurrency probability across voice parts, and a sequence consistency constraint matrix representing the smooth transition probability of single voice parts. The corpus can be a publicly available standard symbolized polyphonic dataset, such as the MAESTRO dataset or the POP909 dataset.

8. A system for anomaly detection and error correction in symbolic polyphonic data structures based on heterogeneous graph neural networks, characterized in that, include: Data parsing module: used to acquire the input symbolic polyphonic data stream, which is a digital polyphonic music event stream; to parse and extract multi-dimensional acoustic feature vectors from the data stream; the multi-dimensional acoustic feature vectors include at least pitch class index, octave cardinality, start timestamp, duration value and voice part index. Heterogeneous graph construction module: used to perform step S1 as described in claim 1, to construct a heterogeneous topology graph containing a double edge structure of Et and Ec; Spatiotemporal feature extraction and anomaly localization module: used to execute steps S2 and S3 as described in claim 1, and output the connected components marked as topological distortion node clusters; State optimization and reconstruction module: used to execute step S4 as described in claim 1, optimize based on global state transition cost and output the reconstructed symbolic polyphonic sequence.

9. An electronic device, characterized in that, It includes at least one processor and a memory communicatively connected to the processor; the memory stores executable instructions to cause the processor to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1-7.

Images