A music segment cross-modal retrieval and infringement positioning method based on an image feature space

Abstract

The application relates to the fields of music copyright protection and cross-modal information retrieval, and discloses a music segment cross-modal retrieval and infringement positioning method based on an image feature space, which comprises the following steps: converting a to-be-detected music segment into a standardized spectrogram, and generating an invariance feature map through time-frequency domain transformation invariance coding; obtaining a local feature sequence and a global feature vector through an image feature extraction network; constructing a hierarchical retrieval architecture, screening Top-K candidate music through coarse retrieval, and detecting infringement start and end time through a timing attention network; and finally outputting positioning results and confidence through double verification of spectral consistency. The application realizes transformation robust coding through a double-branch twin network, realizes timing boundary positioning through a Transformer decoder, considers efficiency and recall rate through dynamic K value hierarchical retrieval, reduces the false detection rate through phase correlation verification, and improves the robustness, accuracy and efficiency of music infringement detection.

Description

Technical Field

[0001] This invention relates to the field of music copyright protection and cross-modal information retrieval, and in particular to a method for cross-modal retrieval and infringement location of music fragments based on image feature space. Background Technology

[0002] Music copyright protection is one of the core issues in the digital content industry. With the rapid development of platforms such as short videos, live streaming, and music streaming, the unauthorized use of music clips is increasing, causing serious economic losses to music copyright holders. Music clip infringement detection and location technology aims to quickly identify copyrighted music similar to the music clip to be detected from a massive music library and accurately output the start and end times of the infringing clip.

[0003] Traditional music retrieval methods primarily rely on audio fingerprinting techniques (such as the Shazam algorithm), which construct hash fingerprints by extracting significant spectral peaks from music and then matching them in a fingerprint database. However, audio fingerprinting methods have limited robustness to audio transformations such as pitch shifting, tempo shifting, noise addition, and compression, and typically can only determine overall similarity, making it difficult to precisely pinpoint the start and end boundaries of infringement. In recent years, with the development of deep learning, converting music into spectrograms and utilizing image feature extraction networks from the field of computer vision for cross-modal retrieval has become a new research direction. Spectrograms present the time, frequency, and energy information of music signals in image form, allowing data originally belonging to the audio modality to be mapped into the image feature space, thereby enabling music retrieval using mature image retrieval techniques. The core challenge of cross-modal retrieval lies in how to eliminate the semantic gap between different modalities and how to achieve a balance between retrieval accuracy and efficiency.

[0004] Existing copyright infringement detection methods for music clips suffer from the following shortcomings: First, audio fingerprinting-based methods are sensitive to audio transformations. Infringers can evade detection through simple techniques such as pitch shifting (raising or lowering by a semitone), speed changes (fast or slow playback), and adding background noise, leading to a significant decrease in recall. Although some methods attempt to extract transformation-robust features (such as Mel-frequency cepstral coefficients, MFCC), these features often lose fine spectral structure information, making it difficult to distinguish between melodious but different music clips. Second, most existing methods only output fragment-level similarity scores, failing to accurately output the start and end times of infringing content within the copyrighted music. In actual copyright disputes, infringers may only use a section of copyrighted music (such as the chorus), resulting in low overall similarity but with the core melody already stolen. Search results lacking boundary localization capabilities are insufficient as valid evidence of infringement. Third, existing cross-modal retrieval methods typically employ a single-layer retrieval architecture. Directly performing fine-grained matching in large-scale music databases involves high computational costs and long response times, making it difficult to meet real-time requirements. Meanwhile, similarity metrics often employ shallow indicators such as cosine similarity or Euclidean distance, lacking effective verification mechanisms for matching results and easily leading to false detections. Fourth, directly using general image feature extraction networks after converting music into spectrograms ignores the time-frequency domain characteristics of the music signal itself, such as the sensitivity of spectrograms to transformations like time-domain translation and frequency scaling, resulting in insufficient feature representation capabilities. Summary of the Invention

[0005] The technical problems to be solved by this invention are: First, how to improve the robustness of music fragment retrieval to audio transformations such as pitch shifting, speed shifting, and noise addition, so that infringers cannot evade detection through simple transformations; Second, how to achieve precise boundary positioning of the start and end times of infringing fragments to meet the requirements for constructing a chain of evidence for copyright infringement; Third, how to improve the retrieval efficiency of large-scale music libraries while ensuring retrieval accuracy, achieving millisecond-level response; Fourth, how to effectively verify the retrieval results, reduce the false detection rate, and improve the credibility of infringement determination.

[0006] Specifically, this invention provides a method for cross-modal retrieval and infringement localization of music fragments based on image feature space, the method comprising the following steps: S1. Obtain the music segment to be detected, convert the music segment to be detected into a spectrogram sequence, and preprocess the spectrogram sequence to obtain a standardized spectrogram; S2. Perform time-frequency domain transform-invariant encoding on the standardized spectrogram to generate a transform-invariant feature map; S3. Input the transformation-invariant feature map into a pre-trained image feature extraction network to extract multi-scale image features, and obtain a local feature sequence of the spectrogram that retains the time dimension and a global feature vector of the spectrogram after global pooling. S4. Construct a hierarchical retrieval architecture, including a coarse retrieval stage and a fine localization stage: In the coarse retrieval stage, the global feature vector of the spectrogram is reduced in dimensionality and then subjected to an approximate nearest neighbor search with the global feature index of the reference music library to select Top-K candidate music and their corresponding reference feature vector sequences and reference spectrogram data, where the value of K is inversely proportional to the length of the music segment to be detected; In the fine localization stage, subsequent localization steps are only performed on the Top-K candidate music. S5. Construct a temporal attention infringement localization network, and use the local feature sequence of the spectrogram and the reference feature vector sequence to perform frame-by-frame infringement boundary detection for each candidate music in the Top-K candidate music, and output the infringement start time and infringement end time. S6. Based on the infringement start time and infringement end time, use the reference spectrogram data to perform dual verification of the infringement location result for spectral consistency, and output the verified infringement location result and infringement confidence score.

[0007] A storage medium storing instructions and data for implementing a method for cross-modal retrieval and infringement location of music fragments based on image feature space.

[0008] A device for cross-modal retrieval and infringement location of music fragments based on image feature space includes: a processor, a storage medium, and an edge computing acceleration module; the processor loads and executes instructions and data in the storage medium to implement a method for cross-modal retrieval and infringement location of music fragments based on image feature space.

[0009] The beneficial effects provided by this invention are as follows: First, it introduces time-frequency domain transform-invariant coding, employing a dual-branch Siamese network using log-Mel scale transformation and constant Q-transform, and pre-training through contrastive learning to make features robust to transformations such as pitch shifting and tempo shifting. Second, it constructs a temporal attention infringement localization network, utilizing a Transformer decoder to calculate cross-modal attention weights between the detected segment and candidate music on the time axis, and accurately outputs the infringement boundary through bidirectional temporal smoothing. Third, it designs a hierarchical retrieval architecture. In the coarse retrieval stage, it uses approximate nearest neighbor search to quickly filter the candidate set, while in the fine localization stage, it performs fine-grained temporal matching only on the candidate set, and dynamically adjusts the K value to be inversely proportional to the segment length, balancing efficiency and recall. Finally, it introduces a dual verification mechanism of spectral consistency, calculating the peak response value of the spectrogram sub-region through phase correlation maps to physically verify the temporal localization results. This invention effectively improves the robustness, accuracy, and efficiency of music segment infringement detection, providing reliable technical support for music copyright protection. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the hardware device operation according to an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0012] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0013] Example 1 Please refer to Figure 1 This invention provides a method for cross-modal retrieval and infringement location of music fragments based on image feature space, comprising the following steps: S1. Obtain the music segment to be detected, convert the music segment to be detected into a spectrogram sequence, and preprocess the spectrogram sequence to obtain a standardized spectrogram; As one embodiment, step S1 is specifically implemented as follows: First, the music segment to be detected is obtained from the audio stream or audio file. This segment can be of any length, such as 3 seconds, 10 seconds, or 30 seconds. If the segment length is less than the preset window length (e.g., 1 second), zero padding or cyclic padding is used to fill the gap. Then, the music segment is framed and windowed: the frame length is set to 2048 sampling points, the frame shift is 512 sampling points, and a Hanning window is used to window each frame. A short-time Fourier transform (STFT) is performed on each frame with 2048 transform points to obtain the power spectrum. The power spectra of all frames are stacked in chronological order to form the original spectrogram, the dimension of which is (number of frequency points × number of time frames). Finally, the original spectrogram is bilinearly interpolated and scaled to a preset size (e.g., 224×224 pixels) and normalized to map the pixel value range to the [0,1] interval to obtain a normalized spectrogram. For example, after the above processing, a 10-second music clip can generate an original spectrogram of approximately (2048 / 2+1=1025 frequency points) × (10×44100 / 512≈861 frames), which can be scaled down to obtain a normalized spectrogram of 224×224. It should be noted that step S1 specifically includes: S11. The music segment to be detected is segmented and windowed. A short-time Fourier transform is performed on each frame to obtain the power spectrum. The power spectra are stacked in time to generate the original spectrogram. S12. Scale the original spectrogram to a preset size and perform normalization processing to obtain a standardized spectrogram; When the length of the music segment to be detected is less than the preset window length, zero padding or cyclic padding is used to fill the gap.

[0014] As one embodiment, the framing and windowing processing in step S1 uses the following parameters: sampling rate Frame length Each sampling point (corresponding to approximately 46.4 milliseconds), frame shift There are 11.6 sampling points (corresponding to approximately 11.6 milliseconds), with an overlap rate of 75% between adjacent frames. The Hanning window function is used, and its expression is:

[0015] After windowing each frame, a short-time Fourier transform is performed, with the number of transform points... The power spectrum was obtained. The power spectra of all frames are stacked time-wise to generate the original spectrogram, which has a size of [size missing]. That is, 1025×N. Then, bilinear interpolation is used to scale the original spectrogram to 224×224 pixels, and max-min normalization is performed on each pixel value:

[0016] A standardized spectrogram is obtained. When the length of the music segment to be detected is less than the preset window length (e.g., less than 1 second, corresponding to about 44 frames), zero padding is used to add zeros to the end of the time axis until the preset window length is met; or cyclic padding is used, that is, the segment content is repeated until the length requirement is met.

[0017] S2. Perform time-frequency domain transform-invariant encoding on the standardized spectrogram to generate a transform-invariant feature map; As one embodiment, step S2 aims to make the extracted features robust to audio transformations such as pitch shifting, speed shifting, and noise addition. Specifically: First, a log-Mel scale transform and a constant Q transform (CQT) are performed on the standardized spectrogram. The log-Mel scale transform converts the linear frequency axis to a Mel frequency axis and takes the logarithmic amplitude to generate a first transformed spectrogram with dimensions of 128 Mel bands × time frames. The constant Q transform uses a non-uniformly spaced frequency resolution, with high resolution for low frequencies and low resolution for high frequencies, to generate a second transformed spectrogram, typically taking 84 octave half-tone bands. Then, the first and second transformed spectrograms are input into a two-branch Siamese network. This Siamese network consists of two subnetworks with identical structures and shared weights, each containing 3 convolutional layers and 2 fully connected layers. Through comparative learning pre-training, the training objective is to ensure that even after transformations such as pitch shifting (e.g., raising the pitch by a semitone), speed change (e.g., 1.2x speed), and noise addition (15dB signal-to-noise ratio), the Euclidean distance between the output feature vectors of homologous music segments is less than a first threshold (e.g., 0.2); while the feature distance between heterologous music segments is greater than a second threshold (e.g., 0.8). Finally, the two feature vectors (e.g., each 512-dimensional) output by the Siamese network are adaptively weighted and fused: first, the spectral energy distribution entropy of the first and second transformed spectrograms is calculated, where the energy of the i-th frequency sub-band is normalized and then the Shannon entropy is calculated. A larger entropy value indicates a greater amount of information in the transformation, and the corresponding weighting coefficient is higher. Let the first feature vector be... The second feature vector is The corresponding entropy value is and Then the fused feature vector is:

[0018] The fused feature vectors are reshaped into a two-dimensional feature map (e.g., 16×16×2-dimensional), resulting in a transformation-invariant feature map.

[0019] It should be noted that step S2 specifically includes: S21. Perform log-Mel scaling transformation and constant Q transformation on the standardized spectrogram to generate a first transformed spectrogram and a second transformed spectrogram. The log-Mel scaling transformation and constant Q transformation respectively generate corresponding frequency sub-band sets. S22. Input the first transformed spectrogram and the second transformed spectrogram into a two-branch twin network. The two-branch twin network is pre-trained through contrastive learning so that the Euclidean distance in the output feature space of the same source music segments after undergoing at least one audio transformation process including pitch shifting, speed shifting, adding background noise, low-pass filtering, and MP3 compression is less than a first threshold, while the feature distance of different source music segments is greater than a second threshold. S23. Adaptively weightedly fuse the first feature vector and the second feature vector output by the dual-branch twin network to generate a transformation-invariant feature map. The weighting coefficients are dynamically calculated based on the spectral energy distribution entropy of the first and second transformed spectrograms. The spectral energy distribution entropy is defined as the Shannon entropy calculated after normalizing the energy of each sub-band in the frequency sub-band set.

[0020] As one embodiment, the log-Mel scaling transformation in step S2 specifically involves converting the linear frequency axis to the Mel frequency axis, using the following formula:

[0021] The frequency bands are uniformly divided into 128 bands along the Mel axis, each corresponding to a different linear frequency range. The power spectral density within each Mel band is then summed and its logarithm taken to obtain the first transformed spectrogram, with dimensions of 128 × number of time frames. The transform parameters for the Constant Q Transform (CQT) are: lowest frequency... (Corresponding to C2 note), highest frequency (Corresponding to C7), each octave contains 12 semitones, total frequency band number The CQT transform kernel is:

[0022] in As a constant factor, it is usually taken as The second transformed spectrogram is obtained, with a dimension of 84 × time frames. The pre-training of the two-branch Siamese network uses a triplet loss function:

[0023] in , Anchor point (original fragment). This is a positive sample (a fragment after homologous transformation). These are negative samples (heterogeneous fragments). After training, the distance between the anchor point and positive samples is ≤0.2, and the distance between the anchor point and negative samples is ≥0.8. The formula for calculating the spectral energy distribution entropy during adaptive weighted fusion is:

[0024] in Let i be the energy of the i-th frequency sub-band. Total number of frequency bands (128 for Mel branch, 84 for CQT branch). Weighting coefficients. , fusion of feature vectors .

[0025] S3. Input the transformation-invariant feature map into a pre-trained image feature extraction network to extract multi-scale image features, and obtain a local feature sequence of the spectrogram that retains the time dimension and a global feature vector of the spectrogram after global pooling. In one embodiment, step S3 uses a ResNet-50 pre-trained on ImageNet as the image feature extraction network. A transform-invariant feature map (224×224×3, expanded to 3 channels by simple channel duplication) is input into the ResNet-50. This network contains 5 convolutional blocks (Conv1 to Conv5) and outputs a multi-scale feature map. This invention uses a Feature Pyramid Network (FPN) structure to fuse the multi-scale features output from Conv3, Conv4, and Conv5, obtaining a fused feature map with dimensions of (temporal dimension × spatial dimension × number of channels). Then, keeping the temporal dimension unchanged (i.e., without global spatial pooling), only average pooling is performed on the spatial dimension to obtain a local spectrogram feature sequence that retains the temporal dimension, for example, with a shape of (number of time frames × 2048 dimensions). Simultaneously, global temporal-spatial average pooling is performed on this feature sequence to obtain a 2048-dimensional global spectrogram feature vector. For example, for an input of 224×224×3, the local feature sequence output by ResNet-50 might be (14×14×2048). After spatial pooling, it becomes (196×2048), and then after temporal compression (preserving the original temporal order), a new local feature sequence is formed. Global pooling yields a 1×2048 global vector. This global vector is used for coarse retrieval, while the local feature sequence is used for subsequent precise localization.

[0026] It should be noted that the pre-trained image feature extraction network in step S3 is a convolutional neural network pre-trained on ImageNet. The multi-scale image features include shallow edge features, mid-level texture features, and deep semantic features. The multi-scale features are fused through a feature pyramid structure to obtain a spectrogram feature map. Before global average pooling, the convolutional neural network outputs a local spectrogram feature sequence that retains the time dimension. This local feature sequence is used for temporal attention infringement localization in step S5. After dimensionality reduction by global average pooling, the local spectrogram feature sequence is used to obtain a global spectrogram feature vector. This global spectrogram feature vector is used for coarse retrieval in step S4.

[0027] As one embodiment, the pre-trained image feature extraction network used in step S3 is ResNet-50, with the following structure: First, a 7×7 convolutional layer (stride 2), followed by batch normalization and ReLU activation, then 3×3 max pooling (stride 2); then sequentially through four residual block groups (Conv2_x to Conv5_x), containing 3, 4, 6, and 3 residual units respectively. Each residual unit consists of 1×1, 3×3, and 1×1 convolutions, and includes skip connections. The Feature Pyramid Network (FPN) fuses the feature maps output from Conv3_x (56×56×256), Conv4_x (28×28×512), and Conv5_x (14×14×2048): first, the high-level features are upsampled by 2x, then added element-wise with the low-level features, and finally fused through a 3×3 convolution. The final output feature map size is 14×14×256.

[0028] To preserve the temporal dimension, this invention does not perform global pooling on the spatial dimension. Instead, it treats the spatial dimension (14×14) as a temporal-spatial hybrid dimension. Since the time axis of the spectrogram corresponds to the horizontal spatial axis of the original input, by keeping the horizontal axis unchanged and performing average pooling on the vertical axis, a local feature sequence of 1×14×256 is obtained. Specifically, let the feature map dimension be... ,in For height (corresponding to the frequency axis). Given the width (corresponding to the time axis), the local feature sequence is: ,get indivual -dimensional vectors. These vectors are then arranged in chronological order to form a local feature sequence. Simultaneously, global average pooling is performed on this feature sequence (for...). and (Averaging the two dimensions) yields a 1×C global feature vector. For example, when At that time, the local feature sequence length is 14, and each vector has 256 dimensions; the global feature vector has 256 dimensions. In practical applications, it can be... Linear interpolation is used to unify the length to a fixed length (e.g., 128) to facilitate subsequent Transformer processing.

[0029] S4. Construct a hierarchical retrieval architecture, including a coarse retrieval stage and a fine positioning stage: In the coarse retrieval stage, the global feature vector of the spectrogram is reduced in dimensionality and then subjected to an approximate nearest neighbor search with the global feature index of the reference music library to select Top-K candidate music and their corresponding reference feature vector sequences and reference spectrogram data. K The value of is inversely proportional to the length of the music segment to be detected; the fine positioning stage only performs the Top- K The candidate music will then undergo subsequent positioning steps. As one embodiment, step S4 enables fast retrieval. First, a global feature index of the reference music library is pre-constructed. The reference music library contains 1 million legal music works, each music is segmented into 2-second segments with a 50% overlap, and a 2048-dimensional reference feature vector is extracted from each segment, and the corresponding reference spectrogram data is stored. The index is constructed using the Hierarchical Navigable Small World Graph (HNSW) algorithm, with the parameters set as follows: M =32, ef_construction =200. In the coarse search stage, the 2048-dimensional global feature vector obtained in step S3 is reduced to 128-dimensional using a Gaussian random projection matrix, and then an approximate nearest neighbor search is performed with the 128-dimensional reference feature vector in the index to calculate the cosine similarity. K The value is dynamically adjusted based on the length of the music segment to be detected: when the segment duration is less than 3 seconds, K =100; when the duration is between 3 and 10 seconds K =50; when the duration is greater than 10 seconds K =20. For example, a 4-second music clip, K =50, meaning the 50 candidate music segments with the highest similarity are selected. Simultaneously, a reference feature vector sequence (for subsequent timeline matching) and reference spectrogram data (for verification) are obtained for each candidate segment. The coarse search stage takes approximately 50 milliseconds to retrieve data from a database of 1 million entries.

[0030] It should be noted that step S4 specifically includes: S41. A global feature index for the reference music library is pre-constructed. The global feature index uses a hierarchical navigable small-world graph algorithm to index the reference feature vector of each reference music in the reference music library. S42. Reduce the dimensionality of the global feature vector of the spectrogram to the same dimension as the global feature index by using a Gaussian random projection matrix, perform an approximate nearest neighbor search, and calculate the cosine similarity between the global feature vector of the spectrogram and each reference feature vector. S43. Dynamically determine based on the duration of the music segment to be detected. K Value: When the duration is less than the first preset value, K Set to 100; when the duration is between the first preset value and the second preset value, K Take 50; when the duration is greater than the second preset value, K Take 20; S44, Output with the highest similarity K Each candidate music segment and its corresponding reference feature vector sequence and reference spectrogram data.

[0031] As one embodiment, the Hierarchical Navigable Small World Graph (HNSW) index construction parameters in step S4 are: number of layers. ,in The maximum number of neighbors per layer is calculated based on the total number of reference feature vectors (approximately 1 million). Dynamic list size during construction Gaussian random projection matrix ,in , The matrix elements are independently and identically distributed in The reduced-dimensional query vector is When performing an approximate nearest neighbor search, cosine similarity is used as the distance metric.

[0032] K The dynamic adjustment strategy for the values ​​is shown in the table below: Table 1 K Value dynamic adjustment strategy

[0033] For example, a short clip of 2.5 seconds, due to its limited information content, requires more candidates to ensure recall. K =100; A 15-second long clip, rich in information. K =20 is sufficient. The output includes: the highest similarity score. K The IDs of the candidate music pieces and the corresponding reference feature vector sequence for each candidate (e.g., each candidate music piece is pre-segmented, and the length of the reference feature vector sequence for each segment is...). L i (2048 dimensions), and reference spectrogram data for each candidate (stored in compressed NumPy array format).

[0034] S5. Construct a temporal attention infringement localization network, and use the local feature sequence of the spectrogram and the reference feature vector sequence to perform frame-by-frame infringement boundary detection for each candidate music in the Top-K candidate music, and output the infringement start time and infringement end time. As one embodiment, step S5 achieves frame-accurate infringement boundary localization. The temporal attention infringement localization network is based on a Transformer decoder structure. The local feature sequence of the spectrogram of the music segment to be detected is denoted as... ,in Let T be the number of time segments (e.g., for a 4-second segment, frame shift 512 points, sampling rate 44.1kHz, T≈344). The reference feature vector sequence of the candidate music segment is denoted as... , This represents the total number of time positions for the candidate music (e.g., approximately 15,000 positions for a 3-minute song). The Transformer decoder employs a 6-layer, 8-attention-head architecture. As a query Calculate the cross-modal attention weight matrix as keys and values. , where each element:

[0035] get Then, a forward-backward smoothing filter is performed on each column (corresponding to the time position of the candidate music): the forward smoothing uses a causal low-pass filter (coefficients...). Backward smoothing employs a non-causal low-pass filter (coefficients) Then, the filtering results from both directions are geometrically averaged to obtain the probability distribution of the starting frame. and end frame probability distribution An adaptive threshold (set to 0.5) is used to segment the distribution and extract continuous intervals. Finally, the start and end regions are paired (the start time must be earlier than the end time), and adjacent or overlapping intervals are merged, outputting the infringement start and end times (in milliseconds). For example, for an infringing piece of music, the output might be a start time of 2.34 seconds and an end time of 7.89 seconds.

[0036] It should be noted that step S5 specifically includes: S51. Obtain the local feature sequence of the spectrogram and preserve it as a time segment sequence along the time axis, with each time segment corresponding to a local feature vector; S52. Obtain the reference feature vector sequence of the candidate music segment, wherein the reference feature vector sequence is pre-stored with a timestamp index; S53. The local feature vector sequence is used as the query input to the Transformer decoder, and the reference feature vector sequence of the candidate music segment is used as the key and value input. The cross-modal attention weight between each time segment and each time position in the candidate music is calculated to obtain the attention weight matrix. S54. Perform forward-backward smoothing filtering on the attention weight matrix along the time axis to generate the start frame probability distribution and the end frame probability distribution. The forward smoothing uses a causal low-pass filter, and the backward smoothing uses a non-causal low-pass filter. The filtering results in the two directions are then geometrically averaged. S55. Adaptive threshold segmentation is applied to the probability distribution of the starting frame and the probability distribution of the ending frame, and continuous intervals exceeding the threshold are extracted as candidate starting regions and candidate ending regions. S56. Pair the candidate start region and the candidate end region and merge the connected components, and output the infringement start time and infringement end time.

[0037] As one embodiment, the Transformer decoder in step S5 adopts a standard architecture, containing 6 decoder layers. Each layer includes: masked multi-head self-attention (the mask ensures that the current position can only depend on previous positions), encoder-decoder cross-attention (using the reference sequence as keys and values), and a feedforward network (FFN). The multi-head attention has 8 heads, and the hidden layer dimension is... FFN inner dimensions The formula for calculating cross-attention is:

[0038] in The output attention weight matrix Size is ,in To query the sequence length, The reference sequence length is used. The specific implementation of the forward-backward smoothing filter: The forward filter uses a first-order recursive low-pass filter.

[0039] Backward filtering from arrive Reverse recursion:

[0040] The final smoothing probability is:

[0041] The probability distribution of the end frame is similar. The adaptive threshold is automatically calculated using the Otsu algorithm, or simply set to a fixed value of 0.5. When merging connected components, if the interval between two intervals is less than 5 frames (approximately 50 milliseconds), they are merged into one interval. The output time is in milliseconds, calculated by multiplying the frame index by the frame shift time (approximately 11.6 milliseconds).

[0042] S6. Based on the infringement start time and infringement end time, use the reference spectrogram data to perform dual verification of the infringement location result for spectral consistency, and output the verified infringement location result and infringement confidence score.

[0043] As one embodiment, step S6 performs fine-grained verification to reduce the false detection rate. First, based on the infringement start and end times output in step S5, a first spectrogram sub-region is extracted from the standardized spectrogram of the music segment to be detected. The size is (frequency count × time frame count), for example, 128 × 300 pixels. Simultaneously, a second spectrogram sub-region of the same time interval is extracted from the reference spectrogram data of the candidate music segment. Then, calculate the local phase correlation maps of the two sub-regions: respectively for and Perform a two-dimensional Fourier transform to obtain and Calculate the cross power spectrum:

[0044] in This indicates element-wise multiplication. Indicates complex conjugation. (For) Performing a two-dimensional inverse Fourier transform yields the phase correlation function matrix. Its peak response value This indicates the degree of consistency between the two sub-regions. A preset verification threshold is set to 0.65. If... If the verification is successful, the start and end times of the infringement will be output. As an infringement confidence score; if If so, the confidence score will be marked as low and a manual review prompt will be triggered. For example, for a genuine infringing segment, It can reach above 0.85; for false positives, It is usually below 0.4.

[0045] It should be noted that step S6 specifically includes: S61. Based on the infringement start time and the infringement end time, extract a first spectrogram sub-region from the standardized spectrogram of the music segment to be detected; S62. Based on the infringement start time and the infringement end time, extract a second spectrogram sub-region from the reference spectrogram data of the candidate music segment; S63. Calculate the local phase correlation spectrum between the first spectrogram sub-region and the second spectrogram sub-region. Specifically, perform two-dimensional Fourier transform on the two sub-regions respectively, calculate the cross power spectrum, and then perform inverse Fourier transform on the cross power spectrum to obtain the phase correlation function matrix. S64. Extract the peak response value of the phase correlation function matrix as the peak correlation. When the peak correlation exceeds the preset verification threshold, the infringement location result is determined to be verified, and the infringement start time, the infringement end time, and the peak correlation are output as the infringement confidence score; otherwise, the infringement confidence score is marked as low confidence and a manual review prompt is triggered.

[0046] As one embodiment, the local phase correlation spectrum calculation process in step S6 is as follows: Let there be two spectrogram sub-regions. and All sizes (For example First, perform a two-dimensional discrete Fourier transform on each sub-region:

[0047] Then calculate the cross-power spectrum:

[0048] in Indicates complex conjugation. (For) The phase correlation function is obtained by performing a two-dimensional inverse Fourier transform:

[0049] exist A peak value appears at a certain point, and the magnitude of the peak value indicates the similarity between the two sub-regions. The peak response value is then recorded. Preset verification threshold .like If the result is correct, the verification passes; otherwise, it fails. For example, for two completely identical spectrogram sub-regions, For unrelated noise, When verification is successful, the infringement start time will be output. End time and confidence score If it fails, set the confidence score to [value]. Alternatively, it may be marked as low confidence and trigger a manual review interface.

[0050] It should be noted that the method also includes step S0: constructing a reference music library, specifically including: S01. Collect a collection of legal music works, segment each music work using a fixed-duration sliding window, with the window duration consistent with the frame length of the segmented processing, and set a preset overlap ratio between adjacent windows to convert each segment into a standardized spectrogram. S02. Perform the same time-frequency domain transform-invariant coding as in step S2 on the standardized spectrogram of each segment to generate a reference transform-invariant feature map; S03. Use the same pre-trained image feature extraction network as in step S3 to extract reference feature vectors, and store the reference spectrogram data corresponding to each segment. Associate each reference feature vector with a timestamp index and record the music time position corresponding to the feature vector. S04. Store the reference feature vector and its timestamp index in the vector database, and store the reference spectrogram data in the associated storage area. Use the hierarchical navigable small world graph algorithm to construct a global feature index to form a reference music library.

[0051] As one embodiment, the process of constructing the reference music library in step S0 is as follows: First, a collection of legal music works is collected, totaling 100,000 complete songs, each with a duration of approximately 3-5 minutes. Each music work is segmented using a sliding window of fixed duration: the window duration is set to 2 seconds, consistent with the aforementioned frame length and frame shift settings (i.e., each window contains approximately 86 frames), and a 50% overlap rate is set between adjacent windows, i.e., a step size of 1 second. For example, a 180-second song will generate approximately 179 segments (each segment 2 seconds). Each segment is independently converted into a standardized spectrogram, using the same conversion method as in step S1, resulting in a 224×224 standardized spectrogram. Then, the same time-frequency domain transform-invariant encoding as in step S2 is performed on the standardized spectrogram of each segment to generate a reference transform-invariant feature map. Next, the same pre-trained ResNet-50 network as in step S3 is used to extract reference feature vectors, while simultaneously storing the original reference spectrogram data corresponding to each segment (scaled to 224×224 and saved in compressed format). A timestamp index is associated with each reference feature vector to record the start and end times (in milliseconds) of that segment in the original music. Finally, the reference feature vectors and their timestamp indices are stored in a vector database (e.g., using the FAISS library), and the reference spectrogram data is stored in an associated storage area (e.g., cloud storage objects). A hierarchical navigable small-world graph algorithm is used to construct a global feature index, with the same index parameters as in claim 5. After construction, the reference music library supports millisecond-level retrieval response. For example, a 4-minute song is divided into approximately 240 segments, each segment extracting 2048-dimensional features, requiring approximately 240×2048×4 bytes ≈ 2MB of storage. Adding the spectrogram data (approximately 240×224×224×4 bytes ≈ 48MB), the overall scale is manageable.

[0052] Example 2: The following is a complete application example: Input: A user submitted a 15-second music clip collected from the Internet, suspecting that it infringed on a certain copyrighted music.

[0053] The processing flow is as follows: 1. The segment sampling rate is 44.1kHz. After framing, windowing, and STFT, the original spectrogram is generated and then scaled and normalized to a 224×224 normalized spectrogram.

[0054] 2. Perform time-frequency domain transform-invariant coding: Generate a 128-dimensional Mel spectrum and an 84-dimensional CQT spectrum, output two 512-dimensional features through a two-branch Siamese network, and obtain a transform-invariant feature map after fusion.

[0055] 3. ResNet-50 feature extraction: obtain local feature sequences (length 14, each 256 dimensions) and global feature vectors (256 dimensions).

[0056] 4. Hierarchical retrieval: Global features are reduced to 128 dimensions for searching within a reference database of 1 million entries. Since the segment duration is 15 seconds (>10 seconds), the segment is selected. K =20, obtaining 20 candidate music segments and their reference feature sequences and spectrograms.

[0057] 5. Temporal Attention Localization: For each candidate, calculate the Transformer cross-attention, and output the infringement boundary after smoothing filtering. Assuming the third candidate is a music match, output the start time of 3.2 seconds and the end time of 14.1 seconds.

[0058] 6. Spectral Consistency Verification: Extract the spectrogram sub-regions of the segment to be detected and candidate music from 3.2-14.1 seconds, and calculate the phase correlation peak. It exceeds the 0.65 threshold and passes the verification.

[0059] 7. Output: Infringement is established with a confidence level of 92%. The infringing segment is located between 3.2 seconds and 14.1 seconds of the copyrighted music.

[0060] Example 3: Please see Figure 2 , Figure 2 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a cross-modal retrieval and infringement location device 401 for music fragments based on image feature space, a processor 402, and a storage medium 403.

[0061] A music fragment cross-modal retrieval and infringement location device 401 based on image feature space: The music fragment cross-modal retrieval and infringement location device 401 based on image feature space implements the music fragment cross-modal retrieval and infringement location method based on image feature space.

[0062] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the method for cross-modal retrieval and infringement location of music fragments based on image feature space.

[0063] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the method for cross-modal retrieval and infringement location of music fragments based on image feature space.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for cross-modal retrieval and infringement location of music fragments based on image feature space, characterized in that: Includes the following steps: S1. Obtain the music segment to be detected, convert the music segment to be detected into a spectrogram sequence, and preprocess the spectrogram sequence to obtain a standardized spectrogram; S2. Perform time-frequency domain transform-invariant encoding on the standardized spectrogram to generate a transform-invariant feature map; S3. Input the transformation-invariant feature map into a pre-trained image feature extraction network to extract multi-scale image features, and obtain a local feature sequence of the spectrogram that retains the time dimension and a global feature vector of the spectrogram after global pooling. S4. Construct a hierarchical retrieval architecture, including a coarse retrieval stage and a fine localization stage: In the coarse retrieval stage, the global feature vector of the spectrogram is reduced in dimensionality and then subjected to an approximate nearest neighbor search with the global feature index of the reference music library to select Top-K candidate music and their corresponding reference feature vector sequences and reference spectrogram data, where the value of K is inversely proportional to the length of the music segment to be detected; In the fine localization stage, subsequent localization steps are only performed on the Top-K candidate music. S5. Construct a temporal attention infringement localization network, and use the local feature sequence of the spectrogram and the reference feature vector sequence to perform frame-by-frame infringement boundary detection for each candidate music in the Top-K candidate music, and output the infringement start time and infringement end time. S6. Based on the infringement start time and infringement end time, use the reference spectrogram data to perform dual verification of the infringement location result for spectral consistency, and output the verified infringement location result and infringement confidence score.

2. The method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in claim 1, characterized in that: Step S1 specifically includes: S11. The music segment to be detected is segmented and windowed. A short-time Fourier transform is performed on each frame to obtain the power spectrum. The power spectra are stacked in time to generate the original spectrogram. S12. Scale the original spectrogram to a preset size and perform normalization processing to obtain a standardized spectrogram; When the length of the music segment to be detected is less than the preset window length, zero padding or cyclic padding is used to fill the gap.

3. The method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in claim 1, characterized in that: Step S2 specifically includes: S21. Perform log-Mel scaling transformation and constant Q transformation on the standardized spectrogram to generate a first transformed spectrogram and a second transformed spectrogram. The log-Mel scaling transformation and constant Q transformation respectively generate corresponding frequency sub-band sets. S22. Input the first transformed spectrogram and the second transformed spectrogram into a two-branch twin network. The two-branch twin network is pre-trained through contrastive learning so that the Euclidean distance in the output feature space of the same source music segments after being processed by at least one audio transformation including pitch shifting, speed shifting, adding background noise, low-pass filtering, and MP3 compression is less than a first threshold, while the feature distance of different source music segments is greater than a second threshold. S23. Adaptively weightedly fuse the first feature vector and the second feature vector output by the dual-branch twin network to generate a transformation-invariant feature map. The weighting coefficients are dynamically calculated based on the spectral energy distribution entropy of the first and second transformed spectrograms. The spectral energy distribution entropy is defined as the Shannon entropy calculated after normalizing the energy of each sub-band in the frequency sub-band set.

4. The method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in claim 1, characterized in that: The pre-trained image feature extraction network in step S3 is a convolutional neural network pre-trained on ImageNet. The multi-scale image features include shallow edge features, mid-level texture features, and deep semantic features. The multi-scale features are fused through a feature pyramid structure to obtain a spectrogram feature map. Before global average pooling, the convolutional neural network outputs a local spectrogram feature sequence that retains the time dimension. This local feature sequence is used for temporal attention infringement localization in step S5. After dimensionality reduction by global average pooling, the local spectrogram feature sequence is used to obtain a global spectrogram feature vector. This global spectrogram feature vector is used for coarse retrieval in step S4.

5. The method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in claim 1, characterized in that: Step S4 specifically includes: S41. A global feature index for the reference music library is pre-constructed. The global feature index uses a hierarchical navigable small-world graph algorithm to index the reference feature vector of each reference music in the reference music library. S42. Reduce the dimensionality of the global feature vector of the spectrogram to the same dimension as the global feature index by using a Gaussian random projection matrix, perform an approximate nearest neighbor search, and calculate the cosine similarity between the global feature vector of the spectrogram and each reference feature vector. S43. Dynamically determine the K value based on the duration of the music segment to be detected: when the duration is less than the first preset value, K is 100; when the duration is between the first preset value and the second preset value, K is 50; when the duration is greater than the second preset value, K is 20. S44. Output the K candidate music segments with the highest similarity and their corresponding reference feature vector sequences and reference spectrogram data.

6. The method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in claim 4, characterized in that: Step S5 specifically includes: S51. Obtain the local feature sequence of the spectrogram and preserve it as a time segment sequence along the time axis, with each time segment corresponding to a local feature vector; S52. Obtain the reference feature vector sequence of the candidate music segment, wherein the reference feature vector sequence is pre-stored with a timestamp index; S53. The local feature vector sequence is used as the query input to the Transformer decoder, and the reference feature vector sequence of the candidate music segment is used as the key and value input. The cross-modal attention weight between each time segment and each time position in the candidate music is calculated to obtain the attention weight matrix. S54. Perform forward-backward smoothing filtering on the attention weight matrix along the time axis to generate the start frame probability distribution and the end frame probability distribution. The forward smoothing uses a causal low-pass filter, and the backward smoothing uses a non-causal low-pass filter. The filtering results in the two directions are then geometrically averaged. S55. Adaptive threshold segmentation is applied to the probability distribution of the starting frame and the probability distribution of the ending frame, and continuous intervals exceeding the threshold are extracted as candidate starting regions and candidate ending regions. S56. Pair the candidate start region and the candidate end region and merge the connected components, and output the infringement start time and infringement end time.

7. The method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in claim 1, characterized in that: Step S6 specifically includes: S61. Based on the infringement start time and the infringement end time, extract a first spectrogram sub-region from the standardized spectrogram of the music segment to be detected; S62. Based on the infringement start time and the infringement end time, extract a second spectrogram sub-region from the reference spectrogram data of the candidate music segment; S63. Calculate the local phase correlation spectrum between the first spectrogram sub-region and the second spectrogram sub-region. Specifically, perform two-dimensional Fourier transform on the two sub-regions respectively, calculate the cross power spectrum, and then perform inverse Fourier transform on the cross power spectrum to obtain the phase correlation function matrix. S64. Extract the peak response value of the phase correlation function matrix as the peak correlation. When the peak correlation exceeds the preset verification threshold, the infringement location result is determined to be verified, and the infringement start time, the infringement end time, and the peak correlation are output as the infringement confidence score; otherwise, the infringement confidence score is marked as low confidence and a manual review prompt is triggered.

8. The method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in claim 1, characterized in that: It also includes step S0: building a reference music library, specifically including: S01. Collect a collection of legal music works, segment each music work using a fixed-duration sliding window, with the window duration consistent with the frame length of the segmented processing, and set a preset overlap ratio between adjacent windows to convert each segment into a standardized spectrogram. S02. Perform the same time-frequency domain transform-invariant coding as in step S2 on the standardized spectrogram of each segment to generate a reference transform-invariant feature map; S03. Use the same pre-trained image feature extraction network as in step S3 to extract reference feature vectors, and store the reference spectrogram data corresponding to each segment. Associate each reference feature vector with a timestamp index and record the music time position corresponding to the feature vector. S04. Store the reference feature vector and its timestamp index in the vector database, and store the reference spectrogram data in the associated storage area. Use the hierarchical navigable small world graph algorithm to construct a global feature index to form a reference music library.

9. A storage medium, characterized in that: The storage medium stores instructions and data to implement the method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in any one of claims 1 to 8.

10. A cross-modal retrieval and infringement location device for music fragments based on image feature space, characterized in that: include: Processor, storage media, and edge computing acceleration modules; The processor loads and executes instructions and data in the storage medium to implement the method for cross-modal retrieval and infringement location of music fragments based on image feature space as described in any one of claims 1 to 8.

Images