Staff identification method and device based on sparrow search algorithm

By combining the quintile image to be detected with the preset image and using the sparrow search algorithm to match the notes and symbol images, the audio similarity evaluation value is generated, and the problem of uncertainty in recognition results caused by the overlap of complex decorative notes and notes is solved, and the accuracy of recognition is improved Sex and efficiency.

CN118918599AActive Publication Date: 2024-11-08GUANGZHOU EVERBRIGHT EDUCATION TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411115402.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-08
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

When identifying the quintile image to be detected, due to complex decorative notes or overlapping notes, and the quintile image to be detected does not exist in the database, it is impossible to determine whether the recognition result of the quintile image to be detected is correct.

Method used

By combining the quintile image to be detected with the preset quintile image, removing the spectral line is divided into notes and symbolic image data sets, the sparrow search algorithm is used to match the most similar standard notes and symbolic images from the database, and audio similarity is generated The evaluation value is determined. If the evaluation value reaches the threshold, the identification will be completed, otherwise the adjustment algorithm parameters will be rematched.

Benefits of technology

It enhances the credibility of the recognition results, improves processing efficiency and accuracy, and ensures the accuracy of the recognition results, especially in complex images and overlapping situations, effectively solves the reliability problem of the recognition results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918599A_ABST
    Figure CN118918599A_ABST
Patent Text Reader

Abstract

The invention discloses a staff identification method and device based on a sparrow search algorithm, and belongs to the technical field of new-generation information. The method comprises the following steps: after a to-be-detected stave image and a preset stave image are combined, removing stave lines in the combined image; classifying and counting a first note image and a first symbol image in the image after the spectral lines are removed; after the first note image and the first symbol image are quantized, matching the most similar standard note image and standard symbol image through a sparrow search algorithm, generating a first audio, acquiring an original audio corresponding to a preset stave image from a database and recording the original audio as a second audio, the audio similarity evaluation value of the first audio and the second audio is calculated, when the audio similarity evaluation value reaches an audio similarity evaluation value threshold value, the current image is selected as a final result, and if the audio similarity evaluation value does not reach the audio similarity evaluation value threshold value, parameters of the sparrow search algorithm are adjusted, and the most similar standard note and standard symbol image are matched again.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of new generation information technology, and in particular to a staff recognition method and device based on a sparrow search algorithm. Background Art

[0002] With the continuous development of computer technology and music art, people's understanding of music creation, dissemination, and performance has also undergone tremendous changes, giving rise to the emerging technology of staff notation recognition. Staff notation recognition technology integrates advanced technologies from multiple fields to automatically identify and extract note information from staff notation.

[0003] Currently, staff notation recognition technology has been preliminarily applied in many fields. It is hoped that in the future it can be combined with knowledge from music theory, signal processing, and other fields to further improve the accuracy and practicality of recognition. Interdisciplinary integration will bring new opportunities and challenges to the development of staff notation recognition technology.

[0004] For example, the invention patent publication with publication number CN110516542B discloses a method and system, storage medium, and terminal for automatically recognizing musical notation images. The method comprises the following steps: identifying paragraph structure information in a musical notation image; combining the musical notation images into a score stream in measure order based on the paragraph structure information; performing semantic segmentation on the musical notation image of each measure to generate a semantic graph of the musical notation unit; inferring a fixed-length vector encoding of the musical notation based on the semantic graph; converting the fixed-length vector encoding into variable-length text; and converting the variable-length text into a MusicXML file. The method and system, storage medium, and terminal for automatically recognizing musical notation images of the present invention can accurately recognize musical notation images and convert them into text, with high recognition accuracy and strong practicality.

[0005] For example, patent publication CN105022993B discloses a method for playing musical notation based on image recognition technology, including: image preprocessing and recognition; removing the musical notation; separating the notes; segmenting the notes in legato; identifying the notes in each measure; and emitting sound according to the corresponding frequencies. This embodiment of the present invention allows users to appreciate the melody of a musical score without understanding the score. The method is low in complexity, easy to implement, and possesses significant practical value.

[0006] However, in the process of implementing the technical solutions of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems:

[0007] When identifying a staff image to be detected, due to the presence of complex decorative notes or overlapping notes, and the staff image to be detected not existing in the database, there is a problem that it is impossible to determine whether the recognition result of the staff image to be detected is correct. Summary of the Invention

[0008] The embodiments of the present application provide a staff recognition method and apparatus based on a sparrow search algorithm, thereby solving the problem of being unable to determine whether the recognition result of the staff image to be detected is correct when identifying the staff image to be detected.

[0009] The embodiment of the present application provides a staff recognition method based on a sparrow search algorithm, comprising the following steps: first, merging a staff image to be detected with a preset staff image, identifying and removing the staff lines in the merged image; identifying each independent image in the image after the staff lines are removed, and dividing it into a first note image data set and a first symbol image data set; after quantizing the images of the first note image and the first symbol image into numerical values, matching the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image from a database through a sparrow search algorithm; obtaining the original audio corresponding to the preset staff image from the database, recording it as the second audio; and recording the preset staff image into the first symbol image. Assume that part of the standard notes and standard symbols corresponding to the staff image are generated as a first audio, and the audio similarity evaluation value between the first audio and the second audio is analyzed; the audio similarity evaluation value is compared with the audio similarity evaluation value threshold; if the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the part of the standard notes and standard symbols corresponding to the staff image to be detected are output; otherwise, the number of iterations and the population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0010] Furthermore, the staff image to be detected is merged with a preset staff image, and the steps of identifying and removing the staff lines in the merged image include: adjusting the notes and symbols in the staff image to be detected and the preset staff image to the same recognition level; merging the preset staff image with the staff image to be detected, which is recorded as: merging the image; analyzing the pixel points in the merged image to find the straight line segments that meet the characteristics of the staff lines; extracting the characteristic information of the staff lines, including position, length, and direction; and setting the pixel value at the position where the staff lines are located as the background color based on the extracted characteristic information.

[0011] Furthermore, each independent image in the image after removing the spectral lines is identified, divided into a first note image data set and a first symbol image data set, and their number is counted separately, including: converting the image after removing the spectral lines into a binary image; identifying independent images in the binary image as connected components; identifying and classifying the connected components as a first note image data set and a first symbol image data set; and counting the number of first note images and first symbol images for subsequent calculations.

[0012] Furthermore, the steps of identifying and classifying connected components into a first note image dataset and a first symbol image dataset include: preparing in advance a template image containing standard staff notes and symbols; matching the connected components with the template image; and classifying the connected components into a first note image dataset and a first symbol image dataset based on the matching results.

[0013] Furthermore, after the images of the first note image and the first symbol image are quantized into numerical values, the steps of matching the most similar note image and the most similar symbol image from the database through the sparrow search algorithm include: quantizing the first note image and the first symbol image into numerical values; obtaining the quantized numerical values ​​of the standard note image and the standard symbol image in the database; and performing similarity matching between the image quantization numerical values ​​of the first note image and the first symbol image and the quantization numerical values ​​of the standard note image and the standard symbol image through the sparrow search algorithm.

[0014] Furthermore, the image quantization values ​​of the first note image and the first symbol image are matched with the quantization values ​​of the standard note image and the standard symbol image by a sparrow search algorithm, and the step of obtaining the most similar standard note image and standard symbol image includes: using the calculation formula of the first note image similarity evaluation value and the first symbol image similarity evaluation value as the objective function, searching the quantitative feature space of the standard note image and the standard symbol image in the database, randomly generating an initial population, each individual in the population representing the quantitative value of the first note image or the first symbol image; calculating the initial first note image similarity evaluation value and the first symbol image similarity evaluation value of the individuals in the population, sorting the individuals in the population from large to small according to the initial first note image similarity evaluation value and the first symbol image similarity evaluation value, and selecting a preset proportion of the individuals at the front of the sequence. The individual acts as a discoverer and the remaining individuals act as joiners; the discoverer is responsible for extensive searching to find the potential maximum first note image similarity evaluation value and first symbol image similarity evaluation value; the joiner is responsible for using the information provided by the discoverer to update and obtain a larger first note image similarity evaluation value and first symbol image similarity evaluation value; traverse and update the position of the population individuals, calculate the image similarity evaluation value of the updated population individuals, and update the population individual type according to the image similarity evaluation value; when the set number of traversals is reached, the traversal ends, and the maximum first note image similarity evaluation value and first symbol image similarity evaluation value are the quantitative values ​​of the standard note image and standard symbol image that are most similar to the first note image and the first symbol image, thereby obtaining the most similar standard note image and standard symbol image to the first note image and the first symbol image.

[0015] Furthermore, the calculation formulas for the first note image similarity evaluation value and the first symbol image similarity evaluation value are as follows:

[0016]

[0017] Where S i represents the similarity evaluation value of the first note image of i, λ represents the weight factor of the note image shape similarity, (1-λ) represents the weight factor of the note image position similarity, j i 、k i 、l i Respectively represent the weights of the note head, stem, and tail of the i-th first note image for the entire first note image, x hi 、x mi 、x ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the i-th note in the first note image, y hi 、y mi 、y ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the standard note image matched with the i-th first note image in the database, x i 、y i represents the quantized value of the image position of the i-th note in the first note image and the quantized value of the standard note image position matched with the i-th first note image in the database, respectively. n represents the nth first symbol image similarity evaluation value, μ represents the weight of the symbol image shape similarity, (1-μ) represents the weight of the symbol image position similarity, m sn 、n sn The image of the nth symbol in the first symbol image and the quantized value of the shape of the standard symbol image in the database that matches the nth first symbol image are represented respectively, m xn 、n xn They respectively represent the image of the nth symbol in the first symbol image and the quantized values ​​of the symbol positions of the standard symbol image matched with the nth first symbol image in the database, where i is the number of the note, i=1,2,3..., N1 represents the number of notes, n is the number of the symbol, n=1,2,3..., N2 represents the number of symbols.

[0018] Furthermore, the steps of generating a first audio using some standard notes and standard symbols corresponding to the preset staff image, and analyzing the audio similarity evaluation value between the first audio and the second audio include: firstly generating the first audio using some standard notes and standard symbols corresponding to the preset staff image, and converting the first audio into a digital audio signal; generating a waveform graph from the digital audio signal, determining the period of the waveform, and calculating the frequency based on the period of the waveform; sampling the digital audio signal to obtain the amplitude value of each sampling point; and using the audio similarity evaluation value calculation formula to calculate the audio similarity evaluation value between the first audio and the second audio.

[0019] Furthermore, the calculation formula for the audio similarity evaluation value is:

[0020]

[0021] Where S represents the audio similarity evaluation value, η represents the weight factor of the time value similarity, (1-η represents the weight factor of the pitch and intensity similarity, j k 、k k Represents the weights of pitch and intensity in the kth sampling point, m lk 、m fk Represents the pitch and intensity value of the kth sampling point in the first audio, m dj Indicates the duration of the jth note in the first audio, n lk 、n fk Represents the pitch and intensity value of the kth sampling point in the second audio, n dj represents the duration of the jth note in the second audio, φ k represents the correction coefficient for the phase difference effect of the kth amplitude value, e represents the base of the natural logarithm, where the pitch is obtained by calculating the frequency of the sound waveform by measuring its period, the intensity is obtained based on the amplitude value of the sampling point, the duration of a single note is calculated based on the whole note duration and the note type, j is the note number, j = 1, 2, 3..., N1 represents the number of notes, k is the number of sampling points, k = 1, 2, 3..., N, N represents the total number of sampling points.

[0022] The embodiment of the present application provides a staff recognition system based on a sparrow search algorithm, which includes: a staff line processing module, a data classification module, an image matching module, a second audio acquisition module, an audio similarity evaluation module, and a final image acquisition module; wherein the staff line processing module is used to merge the staff image to be detected with the preset staff image, identify the staff lines in the merged image and remove them; the data classification module is used to identify each independent image in the image after the staff lines are removed, divide it into a first note image data set and a first symbol image data set, and count their numbers respectively; the image matching module is used to quantize the images of the first note image and the first symbol image into numerical values, and then match the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image from the database through the sparrow search algorithm; the second audio The acquisition module is used to obtain the original audio corresponding to the preset staff image from the database, recorded as the second audio; the audio similarity evaluation module is used to generate the first audio from the part of the standard notes and standard symbols corresponding to the preset staff image, and analyze the audio similarity evaluation value between the first audio and the second audio; the final image acquisition module is used to compare the audio similarity evaluation value with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed and the part of the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and the population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. By converting the selected images into audio and comparing them with the audio portion of a preset staff image, the accuracy of image recognition is verified by referring to the audio similarity evaluation value, further enhancing the credibility of the results and effectively solving the problem in the existing technology of being unable to determine whether the recognition result of the staff image to be detected is correct.

[0025] 2. The entire process is highly automated, from image merging, spectral line removal, object recognition and classification to feature quantification matching, and then to audio generation and comparison, which reduces manual intervention and improves processing efficiency and accuracy.

[0026] 3. By processing images and optimizing similarity evaluation values ​​using the sparrow search algorithm, the notes and symbols in the image can be accurately identified and the most similar image can be matched. When the audio similarity evaluation value does not reach the audio similarity evaluation value threshold, the most similar standard note image and standard symbol image can be re-matched by adjusting the number of iterations and population size parameters of the algorithm until the audio similarity evaluation value reaches the audio similarity evaluation value threshold, thereby ensuring the accuracy of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of a staff notation recognition method based on a sparrow search algorithm provided in an embodiment of the present application;

[0028] Figure 2 A schematic structural diagram of a staff notation recognition system based on a sparrow search algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The present invention provides a staff recognition method and device based on a sparrow search algorithm to solve the problem of uncertainty in determining the correctness of the recognition result of a staff image to be detected due to the presence of complex ornamental notes or overlapping notes, and the fact that the staff image to be detected does not exist in a database. The staff image to be detected is merged with a preset staff image, and the image is separated into a first note image and a first symbol image after removing the stave lines. The most similar standard note image and standard symbol image are matched using a sparrow search algorithm to generate a first audio signal. The original audio corresponding to the preset staff image is obtained from a database and recorded as the second audio signal. An audio similarity evaluation value is calculated between the first and second audio signals, and the audio similarity evaluation value is compared with an audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, recognition is completed, and the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image are rematched until the audio similarity evaluation value reaches the audio similarity evaluation value threshold.

[0030] The technical solution in the embodiment of the present application solves the problem that when identifying the staff image to be detected, due to the presence of complex decorative notes or note overlap, and the staff image to be detected does not exist in the database, it is impossible to determine whether the recognition result of the staff image to be detected is correct. The overall idea is as follows: first, the staff image to be detected is merged with the preset staff image, and the staff lines in the merged image are identified and removed; each independent image in the image after the staff lines are removed is identified and divided into a first note image data set and a first symbol image data set; after quantizing the images of the first note image and the first symbol image into numerical values, the most similar standard note image and standard symbol corresponding one-to-one to the first note image and the first symbol image are matched from the database through the sparrow search algorithm. image; obtaining the original audio corresponding to the preset staff image from the database, recorded as the second audio; generating the first audio with part of the standard notes and standard symbols corresponding to the preset staff image, and analyzing the audio similarity evaluation value between the first audio and the second audio; comparing the audio similarity evaluation value with the audio similarity evaluation value threshold; if the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the part of the standard notes and standard symbols corresponding to the staff image to be detected are output; otherwise, the number of iterations and the population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0031] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] like Figure 1 FIG. 1 is a flow chart of a staff notation recognition method based on a sparrow search algorithm according to an embodiment of the present application. The method is applied to a staff notation recognition device based on a sparrow search algorithm. The method includes the following steps:

[0033] First, the staff image to be detected is merged with the preset staff image, and the staff lines in the merged image are identified and removed; each independent image in the image after the staff lines are removed is identified and divided into a first note image data set and a first symbol image data set; after the images of the first note image and the first symbol image are quantized into numerical values, the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are matched from the database through the sparrow search algorithm; the original audio corresponding to the preset staff image is obtained from the database and recorded as the second audio; some standard notes corresponding to the preset staff image are matched and a standard symbol to generate a first audio, analyze the audio similarity evaluation value between the first audio and the second audio; compare the audio similarity evaluation value with the audio similarity evaluation value threshold; if the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the standard notes and standard symbols corresponding to the staff image to be detected are output; otherwise, the number of iterations and the population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0034] Furthermore, the staff image to be detected is merged with the preset staff image, and the steps of identifying and removing the staff lines in the merged image include: adjusting the notes and symbols in the staff image to be detected and the preset staff image to the same recognition level; merging the two images, the preset staff image and the staff image to be detected, and recording them as a merged image; merging the preset staff image with the staff image to be detected includes a variety of methods, such as simple image splicing, so that the key notes and symbols maintain a uniform recognition level, or dividing the preset staff image into a single or multiple notes and symbols and interspersing them in the preset staff image, etc.; analyzing the pixel points in the merged image to find the straight line segments that meet the characteristics of the staff lines; extracting the characteristic information of the staff lines, including position, length, and direction; and setting the pixel value of the position where the staff lines are located as the background color according to the extracted characteristic information.

[0035] In this embodiment, image merging can be a merging of the entire image, or it can be a merging of the images after splitting them into units of musical measures. The merging order can also be multiple. The preset staff image can be divided into multiple parts and interspersed with different positions in the staff image to be detected, or the preset staff image can be placed in any orientation or position of the staff image to be detected. That is, in order to better integrate the preset staff image and the staff image to be detected, the preset staff image after recognition can be used as a judgment basis for verifying the recognition accuracy. The same recognition level indicates that the notes and symbols of the main parts of the two images belong to the same size level or resolution level. The size level can be pre-determined based on the size of the image and the size of the notes and symbols. For example, the size grade can be divided into 1, 2, 3, 4... grades. The first grade is the smallest size grade, specifically less than 1 mm, the second grade is 1-2 mm, and so on. This grading standard can make the image recognition process at a similar size to avoid recognition errors caused by different sizes. Similarly, the consistency of resolution level means that the resolution of the notes or symbols in the two pictures is adjusted to a unified level, for example, the clarity is adjusted to a consistent level, or the pixels are adjusted to a consistent level. In addition, if the clarity or pixels of the picture are poor, image restoration technology can be used in advance to repair or increase the clarity, fill in the blank image, and then adjust it. This operation can make it possible to complete the recognition even if the image to be processed is poor. When identifying musical notation, the lines of the score often cross or overlap with most musical symbols, which destroys the unique features of the symbols and makes subsequent symbol recognition difficult. Deleting the lines can reduce recognition interference, make the features of the note symbols more prominent, and help improve the accuracy and efficiency of feature extraction; it can also simplify the processing flow of the musical notation image, reduce unnecessary calculations, and increase the recognition speed accordingly; in addition, the existence of the lines may cause the recognition system to mistakenly regard the lines as part of the musical symbols, resulting in recognition errors. Deleting the lines can enable the system to more accurately identify the musical symbols in the score, including notes, rests, dynamic marks, etc., which significantly improves the overall accuracy of musical notation recognition.

[0036] Furthermore, each independent image in the image after removing the spectral lines is identified, divided into a first note image data set and a first symbol image data set, and their number is counted separately, including: converting the image after removing the spectral lines into a binary image so that the pixels in the image have only two possibilities, 0 and 255; identifying independent images in the binary image as connected components; identifying and classifying the connected components as a first note image data set and a first symbol image data set; and counting the number of first note images and first symbol images for subsequent calculations.

[0037] In this embodiment, independent images are segmented based on whether the first note image or the first symbol image is connected. For example, triplets, quadruplets, and other notes are treated as a whole as one note data. When the image after removing the spectral lines is segmented into independent images, it is first necessary to traverse each pixel point of the binary image and assign the same label to the pixels belonging to the same connected component based on the value of the pixel point and the relationship between the adjacent pixels.

[0038] Furthermore, the steps of identifying and classifying connected components into a first note image dataset and a first symbol image dataset include: preparing in advance a template image containing standard staff notes and symbols; matching the connected components with the template image; and classifying the connected components into a first note image dataset and a first symbol image dataset based on the matching results.

[0039] In this embodiment, when matching the connected component with the template image, possible note and symbol areas are screened out based on the size, shape and other features of the connected component, the morphological features of the connected component and the template image are extracted, and a suitable data set is selected for matching. Based on the matched data set, the connected component is divided into a first note image data set and a first symbol image data set. In addition, when the image contains multiple notes and symbols and there are specific spatial relationships or layout rules between them, contextual relevance is used to improve the accuracy of classification.

[0040] Furthermore, after the images of the first note image and the first symbol image are quantized into numerical values, the steps of matching the most similar note image and the most similar symbol image from the database through the sparrow search algorithm include: first quantizing the first note image and the first symbol image into numerical values; obtaining the quantized numerical values ​​of the standard note image and the standard symbol image in the database; and performing similarity matching between the image quantization numerical values ​​of the first note image and the first symbol image and the quantization numerical values ​​of the standard note image and the standard symbol image through the sparrow search algorithm.

[0041] In this embodiment, this method has important benefits and practical application value in music image processing and digital archive management: by calculating the similarity with existing staff images, images similar to specific staffs can be retrieved more accurately from a large number of databases, which is very useful for music scholars, educators and database managers, and can quickly locate the required materials; automated image similarity evaluation and retrieval can greatly improve work efficiency, especially when processing large-scale databases. This method reduces the need for manual retrieval and saves a lot of time and labor resources; at the same time, it allows users to select from similar images, providing greater flexibility and personalized options. Users can choose the image that best meets their expectations according to their specific needs, enhancing user experience experience and satisfaction; in music archives and digital library projects, it is crucial to be able to effectively organize and retrieve image materials. The method can help institutions better manage their image materials and improve the accessibility and usability of data; scholars and students can also use the method to find specific staffs to support research and teaching activities related to musicology, history or art, such as studying the historical evolution of different versions of staffs, or analyzing the style of a composer's works; by providing multiple images similar to specific staffs, it can also inspire users to have new insights and creative thinking about different musical works; in addition, the implementation of image similarity retrieval technology can promote the development of related technologies, such as image processing algorithms, machine learning models and user interface design, further improving the overall technical level and application scope.

[0042] Furthermore, the image quantization values ​​of the first note image and the first symbol image are matched with the quantization values ​​of the standard note image and the standard symbol image by a sparrow search algorithm, and the step of obtaining the most similar standard note image and standard symbol image includes: using the calculation formula of the first note image similarity evaluation value and the first symbol image similarity evaluation value as the objective function, searching the quantitative feature space of the standard note image and the standard symbol image in the database, randomly generating an initial population, each individual in the population representing the quantitative value of the first note image or the first symbol image; calculating the initial first note image similarity evaluation value and the first symbol image similarity evaluation value of the individuals in the population, sorting the individuals in the population from large to small according to the initial first note image similarity evaluation value and the first symbol image similarity evaluation value, and selecting a preset proportion of the individuals at the front of the sequence. The individual acts as a discoverer and the remaining individuals act as joiners; the discoverer is responsible for extensive searching to find the potential maximum first note image similarity evaluation value and first symbol image similarity evaluation value; the joiner is responsible for using the information provided by the discoverer to update and obtain a larger first note image similarity evaluation value and first symbol image similarity evaluation value; traverse and update the position of the population individuals, calculate the image similarity evaluation value of the updated population individuals, and update the population individual type according to the image similarity evaluation value; when the set number of traversals is reached, the traversal ends, and the maximum first note image similarity evaluation value and first symbol image similarity evaluation value are the quantitative values ​​of the standard note image and standard symbol image that are most similar to the first note image and the first symbol image, thereby obtaining the most similar standard note image and standard symbol image to the first note image and the first symbol image.

[0043] In this embodiment, the sparrow search algorithm optimizes and solves the problem by simulating the process of sparrows searching for food in nature. Through the behaviors of foraging, flocking, jumping, and escaping, it is efficient and accurate in solving the optimization problem. The sparrow search algorithm has a simple structure, fast convergence speed, and strong global search capability, which is conducive to quickly optimizing the similarity of staff images.

[0044] Furthermore, the calculation formulas for the first note image similarity evaluation value and the first symbol image similarity evaluation value are as follows:

[0045]

[0046] Where S i represents the similarity evaluation value of the first note image of i, λ represents the weight factor of the note image shape similarity, (1-λ) represents the weight factor of the note image position similarity, j i 、k i 、l i Respectively represent the weights of the note head, stem, and tail of the i-th first note image for the entire first note image, x hi、x mi 、x ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the i-th note in the first note image, y hi 、y mi 、y ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the standard note image matched with the i-th first note image in the database, x i 、y i represents the quantized value of the image position of the i-th note in the first note image and the quantized value of the standard note image position matched with the i-th first note image in the database, respectively. n represents the nth first symbol image similarity evaluation value, μ represents the weight of the symbol image shape similarity, (1-μ) represents the weight of the symbol image position similarity, m sn 、n sn The image of the nth symbol in the first symbol image and the quantized value of the shape of the standard symbol image in the database that matches the nth first symbol image are represented respectively, m xn 、n xn represents the image of the nth symbol in the first symbol image and the quantized value of the symbol position of the standard symbol image matched with the nth first symbol image in the database, i is the number of the note, i=1,2,3..., N1 represents the number of notes, n is the number of the symbol, n=1,2,3..., N2 represents the number of symbols, S 总 Represents the total similarity evaluation value.

[0047] The total similarity evaluation value formula calculates the similarity between the total merged image and the composite image of the recognized standard musical note image and the standard symbol image. If the most similar standard musical note image and standard symbol image that are matched one-to-one with the first musical note image and the first symbol image are separated from the staff image to be detected and the preset staff image, and the most similar standard musical note image and standard symbol image that are matched one-to-one with the first musical note image and the first symbol image are combined into a second preset staff image, through S 总 The total similarity evaluation value between the second preset staff image and the preset staff diagram is calculated using the formula, which can indirectly reflect the accuracy of the recognition result of the staff image to be detected.

[0048] In this embodiment, the images and positions of notes and symbols are quantized, which involves the comparison of image features and the precise representation of spatial positions, wherein the image quantization method includes: first determining the range of quantization values, which determines how many different levels the images of notes and symbols will be divided into, traversing each note image and symbol image, and splitting the note image and symbol image into a fixed number of characteristic shapes according to the size and shape of the allocated area of ​​the pixel points of each note image and symbol image, and assigning a quantization value to the shape feature of each note image and symbol image, a quantization value represents a fixed characteristic shape, and the closer the quantization value, the more similar the characteristic shapes; combining the quantization values ​​into a feature vector, which can represent the quantized image of a single note image or symbol image; position quantization generally refers to the precise representation of the spatial coordinates of objects or feature points in an image. In this embodiment, the position quantization of the note image and symbol image refers to the position of each note image and symbol image relative to the original position of the image. The coordinate values ​​of the staff line positions are determined by re-adding the staff lines before removing them or after splitting them into individual images. This is used to determine the position of each note and the corresponding staff line. For notes on the staff lines, a coordinate value is set for each line and space in the staff line. The coordinate value is the quantized value of the position of the note and symbol, such as the coordinate value of the note on the third line is 3, the coordinate value of the note on the third space is 3.5, and so on. When a note exceeds the range of the staff line, a lead-in line is used to locate its position, and a coordinate value is set for each lead-in line and space. For example, the coordinate value of the first lead-in line above the staff line is 6, and the coordinate value of the first lead-in line below the staff line is 0. Various symbols such as clefs, key signatures, and time signatures often appear in the staff. The positions of these symbols are usually fixed. For example, the treble clef is located above the second line, and the coordinate value is calculated based on its position. For other special symbols such as dots and slurs, values ​​are set according to their positions on the notes, thereby achieving quantization of the positions of the note images and symbol images.

[0049] Furthermore, the steps of generating a first audio using some standard notes and standard symbols corresponding to the preset staff image, and analyzing the audio similarity evaluation value between the first audio and the second audio include: firstly generating the first audio using some standard notes and standard symbols corresponding to the preset staff image, and converting it into a digital audio signal; displaying the digital audio signal as a waveform graph, specifically generating a corresponding audio waveform through analog audio software, or inputting it into a device that can automatically analyze sound signals to obtain a corresponding waveform, determine the period of the waveform, and calculate the frequency according to the period of the waveform; sampling the digital audio signal to obtain the amplitude value of each sampling point; and using the audio similarity evaluation value calculation formula to calculate the audio similarity evaluation value between the first audio and the second audio.

[0050] In this embodiment, a method for evaluating the similarity between generated audio and the audio portion corresponding to a stave image through digital processing is described, wherein the formula for calculating the frequency based on the period of the waveform is:

[0051] Where f represents the frequency of the waveform, and T represents the period of the waveform.

[0052] Furthermore, the calculation formula for the audio similarity evaluation value is:

[0053]

[0054] Where S represents the audio similarity evaluation value, η represents the weight factor of the time value similarity, (1-η represents the weight factor of the pitch and intensity similarity, j k 、k k Represents the weights of pitch and intensity in the kth sampling point, m lk 、m fk Represents the pitch and intensity value of the kth sampling point in the first audio, m dj Indicates the duration of the jth note in the first audio, n lk 、n fk Represents the pitch and intensity value of the kth sampling point in the second audio, n dj represents the duration of the jth note in the second audio, φ k represents the correction coefficient for the phase difference effect of the kth amplitude value, e represents the base of the natural logarithm, where the pitch is obtained by calculating the frequency of the sound waveform by measuring its period, the intensity is obtained based on the amplitude value of the sampling point, the duration of a single note is calculated based on the whole note duration and the note type, j is the note number, j = 1, 2, 3..., N1 represents the number of notes, k is the number of sampling points, k = 1, 2, 3..., N, N represents the total number of sampling points.

[0055] In this embodiment, the audio similarity evaluation value calculation formula is mainly used to quantify the similarity between two or more audio signals. It can be calculated based on the different characteristics of the audio signals to obtain a similarity score that directly reflects the similarity or difference between the audio signals. In applications such as audio search, recommendation, and copyright detection, the audio similarity evaluation value calculation formula can significantly improve processing efficiency, quickly find similar audio files or determine the copyright ownership of audio; in applications such as music recommendation and noise cancellation, the application of the audio similarity evaluation value calculation formula can provide users with a more personalized and high-quality audio experience; with the continuous development of technologies such as deep learning, the audio similarity evaluation value calculation formula is also constantly innovating and optimizing, bringing more technological breakthroughs and application innovations to the field of audio processing.

[0056] Furthermore, the system includes: a spectral line processing module, a data classification module, an image matching module, a second audio acquisition module, an audio similarity evaluation module, and a final image acquisition module; wherein the spectral line processing module is used to merge the staff image to be detected with the preset staff image, identify the spectral lines in the merged image and remove them; the data classification module is used to identify each independent image in the image after the spectral lines are removed, divide it into a first note image data set and a first symbol image data set, and count their numbers respectively; the image matching module is used to quantize the images of the first note image and the first symbol image into numerical values, and then match the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image from the database through a sparrow search algorithm; the second audio acquisition module is used to obtain the original audio corresponding to the preset staff image from the database, recorded as The second audio; the audio similarity evaluation module is used to generate the first audio from the part of the standard notes and standard symbols corresponding to the preset staff image, and analyze the audio similarity evaluation value between the first audio and the second audio; the final image acquisition module is used to compare the audio similarity evaluation value with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the part of the standard notes and standard symbols corresponding to the staff image to be detected are output; otherwise, by adjusting the number of iterations and the population size parameter of the sparrow search algorithm, the standard notes and standard symbols in the database are added, and the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, wherein adjusting the population size parameter is to adjust the number of discoverers and joiners.

[0057] like Figure 2 As shown, it is a structural diagram of the staff notation recognition system based on the sparrow search algorithm provided in an embodiment of the present application. The staff notation recognition system based on the sparrow search algorithm provided in an embodiment of the present application includes: a staff line processing module, a data classification module, an image matching module, a second audio acquisition module, an audio similarity evaluation module, and a final image acquisition module.

[0058] In summary, the embodiment of the present application merges the image of the to-be-detected staff and the preset staff image, and then removes the staff lines in the merged image; classifies and counts the first note image and the first symbol image in the image after the staff lines are removed; after quantifying the first note image and the first symbol image, the most similar standard note image and standard symbol image are matched through the sparrow search algorithm to generate the first audio, obtain the original audio corresponding to the preset staff image from the database, record it as the second audio, calculate the audio similarity evaluation value of the first audio and the second audio, and when the audio similarity evaluation value reaches the audio similarity evaluation value threshold, select the current image as the final result. If the audio similarity evaluation value threshold is not reached, adjust the parameters of the sparrow search algorithm and re-match the most similar standard note and standard symbol image.

[0059] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0063] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0064] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A five-line score recognition method based on a sparrow search algorithm is characterized by: The method comprises the following steps: Merge the stave image to be detected with the preset stave image, identify and remove the stave lines in the merged image; identifying each independent image in the image after the staves are removed, and dividing it into a first musical note image dataset and a first symbol image dataset; After quantizing the first musical note image and the first symbol image into numerical values, the most similar standard musical note image and standard symbol image corresponding to the first musical note image and the first symbol image are matched from a database by a sparrow search algorithm; Obtaining the original audio corresponding to the preset stave image from the database, recorded as the second audio; Generate a first audio using part of the standard notes and standard symbols corresponding to the preset stave image, and analyze an audio similarity evaluation value between the first audio and the second audio; The audio similarity evaluation value is compared with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

2. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: The step of merging the to-be-detected stave image with the preset stave image, and identifying and removing the staves in the merged image comprises: Adjusting the notes and symbols in the to-be-detected stave image and the preset stave image to the same recognition level; Merge the preset stave image with the stave image to be detected, and record it as: merged image; Analyze the pixels in the merged image and find the straight line segments that meet the spectral line characteristics; Extract characteristic information of spectral lines, including position, length, and direction; According to the extracted feature information, the pixel value at the location of the spectrum line is set as the background color.

3. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: The steps of identifying each independent image in the image after the spectrum is removed, and dividing it into a first musical note image data set and a first symbol image data set, specifically include: Convert the image after removing the spectral lines into a binary image; Identify independent images in a binary image as connected components; identifying and classifying connected components as a first musical note image dataset and a first symbol image dataset; The number of the first musical note images and the first symbol images is counted.

4. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 3, characterized in that: The steps of identifying and classifying the connected components as the first musical note image dataset and the first symbol image dataset include: Prepare template images containing standard stave notes and symbols in advance; Perform matching operation on the connected components and the template image; According to the matching result, the connected components are classified into a first musical note image dataset and a first symbol image dataset.

5. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: After quantizing the first musical note image and the first symbol image into numerical values, the step of matching the most similar musical note image and the most similar symbol image from the database by using a sparrow search algorithm comprises: quantizing the first musical note image and the first symbol image into numerical values; Obtain quantized values ​​of standard musical note images and standard symbol images in the database; The image quantization values ​​of the first note image and the first symbol image are similarly matched with the quantization values ​​of the standard note image and the standard symbol image by using the sparrow search algorithm to obtain the most similar standard note image and standard symbol image.

6. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 5, characterized in that: The step of performing similarity matching between the image quantization values ​​of the first note image and the first symbol image and the quantization values ​​of the standard note image and the standard symbol image by using the sparrow search algorithm to obtain the most similar standard note image and standard symbol image comprises: The calculation formula of the similarity evaluation value of the first note image and the first symbol image is used as the objective function, the search range is the quantitative feature space of the standard note image and the standard symbol image in the database, and an initial population is randomly generated, each individual in the population represents the quantitative value of the first note image or the first symbol image; Calculate the initial first note image similarity evaluation value and the first symbol image similarity evaluation value of the population individuals, sort the population individuals from large to small according to the initial first note image similarity evaluation value and the first symbol image similarity evaluation value, select a preset proportion of individuals at the front of the sequence as discoverers, and the remaining individuals as joiners; The finder is responsible for extensive searching to find the potential maximum first note image similarity evaluation value and the first symbol image similarity evaluation value; The joiner is responsible for updating the information provided by the finder to obtain a larger first note image similarity evaluation value and first symbol image similarity evaluation value; Traverse and update the positions of the individuals in the population, calculate the image similarity evaluation values ​​of the updated individuals in the population, and update the types of the individuals in the population according to the image similarity evaluation values; The traversal ends when the set number of traversals is reached. The maximum first note image similarity evaluation value and the first symbol image similarity evaluation value are the quantized values ​​of the standard note image and the standard symbol image that are most similar to the first note image and the first symbol image, thereby obtaining the most similar standard note image and the standard symbol image to the first note image and the first symbol image.

7. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 6, characterized in that: The calculation formula of the first note image similarity evaluation value and the first symbol image similarity evaluation value is: In the formula, S i represents the similarity evaluation value of the first note image of the i-th image, λ represents the weight factor of the note image shape similarity, (1-λ) represents the weight factor of the note image position similarity, j i , k i , l i Respectively represent the weights of the note head, stem, and tail of the i-th first note image for the entire first note image note, x hi 、x mi 、x ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the i-th note in the first note image, y hi ,y mi ,y ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the standard note image matched with the i-th first note image in the database, x i ,y i represents the quantized value of the image position of the i-th note in the first note image and the quantized value of the standard note image position matched with the i-th first note image in the database, respectively. n represents the similarity evaluation value of the nth first symbol image, μ represents the weight of the symbol image shape similarity, (1-μ) represents the weight of the symbol image position similarity, m sn 、n sn The image of the nth symbol in the first symbol image and the quantized value of the shape of the standard symbol image in the database that matches the nth first symbol image are represented respectively, and m xn 、n xn They respectively represent the image of the nth symbol in the first symbol image and the quantized values ​​of the symbol positions of the standard symbol image in the database that matches the nth first symbol image. When i is the number of the note, i=1,2,3..., N1 represents the number of the note, n is the number of the symbol, n=1,2,3..., N2 represents the number of the symbol.

8. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: The step of generating a first audio with part of the standard notes and standard symbols corresponding to the preset stave image, and analyzing the audio similarity evaluation value between the first audio and the second audio comprises: Firstly, some standard notes and standard symbols corresponding to the preset stave image are used to generate a first audio, and the first audio is converted into a digital audio signal; Generate a waveform diagram of the digital audio signal, determine the period of the waveform, and calculate the frequency based on the period of the waveform; Sampling the digital audio signal to obtain the amplitude value of each sampling point; The audio similarity evaluation value calculation formula is used to calculate the audio similarity evaluation value of the first audio and the second audio.

9. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 8, characterized in that: The calculation formula of the audio similarity evaluation value is: Where S represents the audio similarity evaluation value, η represents the weight factor of the time value similarity, (1-η) represents the weight factor of the pitch and intensity similarity, and j k , k k Represent the weights of pitch and intensity at the kth sampling point, m lk 、m fk Respectively represent the pitch and intensity value of the kth sampling point in the first audio, m dj Indicates the duration of the jth note in the first audio, n lk 、n fk Represents the pitch and intensity value of the kth sampling point in the second audio, n dj represents the duration of the jth note in the second audio, φ k It represents the correction coefficient of the phase difference of the k-th amplitude value, e represents the base of the natural logarithm, j represents the number of the note, j=1,2,3..., N1 represents the number of notes, k represents the number of sampling points, k=1,2,3..., N, N represents the total number of sampling points.

10. The five-line score recognition system based on the sparrow search algorithm is characterized by: include: Spectrum line processing module, data classification module, image matching module, second audio acquisition module, audio similarity evaluation module, final image acquisition module; The spectral line processing module is used to merge the to-be-detected stave image with the preset stave image, identify the spectral lines in the merged image and remove them; The data classification module is used to identify each independent image in the image after the spectrum is removed, divide it into a first musical note image data set and a first symbol image data set, and count their numbers respectively; The image matching module is used to quantize the first musical note image and the first symbol image into numerical values, and then match the most similar standard musical note image and standard symbol image that correspond one-to-one to the first musical note image and the first symbol image from a database through a sparrow search algorithm; The second audio acquisition module is used to acquire the original audio corresponding to the preset stave image from the database, which is recorded as the second audio; The audio similarity evaluation module is used to generate a first audio from some standard notes and standard symbols corresponding to the preset stave image, and analyze the audio similarity evaluation value between the first audio and the second audio; The final image acquisition module is used to compare the audio similarity evaluation value with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed and the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

Citation Information

Patent Citations

  • Method for converting musical notation image into sound file

    CN101661745A

  • Stave image recognition and encoding method and system based on real-time video stream

    CN108664977A

  • Coal rock boundary image enhancement method based on chaos sparrow search algorithm

    CN113344828A