Method and device for staff recognition based on sparrow search algorithm

Abstract

The present invention discloses a five-line staff recognition method and device based on a sparrow search algorithm, which belongs to the field of new generation information technology. The method comprises the following steps: after merging the to-be-detected and preset five-line staff images, removing the staves in the merged image; classifying and counting the first note image and the first symbol image in the image after the staves are removed; after quantifying the first note image and the first symbol image, matching the most similar standard note image and standard symbol image through a sparrow search algorithm, generating a first audio, obtaining the original audio corresponding to the preset five-line staff image from a database as the second audio, calculating the audio similarity evaluation value of the first audio and the second audio, and when the audio similarity evaluation value reaches the audio similarity evaluation value threshold, selecting the current image as the final result, and if the audio similarity evaluation value threshold is not reached, adjusting the parameters of the sparrow search algorithm, and re-matching the most similar standard note and standard symbol image.

Description

Technical Field

[0001] The present invention relates to the field of new generation information technology, and in particular to a five-line music score recognition method and device based on a sparrow search algorithm. Background Art

[0002] With the continuous development of computer technology and music art, people's understanding of music creation, dissemination and performance channels has also undergone tremendous changes, resulting in the emergence of a new technology called staff recognition. Staff recognition technology integrates advanced technologies from multiple fields to automatically recognize and extract note information from staff.

[0003] At present, staff notation recognition technology has been preliminarily applied in many fields. It is expected that in the future it can be combined with knowledge of music theory, signal processing and other fields to further improve the accuracy and practicality of recognition. Interdisciplinary integration will bring new opportunities and challenges to the development of staff notation recognition technology.

[0004] For example, the invention patent with announcement number CN110516542B announces a method and system, storage medium and terminal for automatic recognition of stave images, including: identifying paragraph structure information in stave images, combining the stave images into a score stream in bar order according to the paragraph structure information; semantically segmenting the score images of each bar to generate a semantic graph of the score unit; inferring the fixed-length vector encoding of the score based on the semantic graph; converting the fixed-length vector encoding into a text of indefinite length; and converting the text of indefinite length into a MusicXML file. The method and system, storage medium and terminal for automatic recognition of stave images of the present invention can accurately recognize music score images and realize textualization, with high recognition accuracy and strong practicality.

[0005] For example, the invention patent announcement with announcement number: CN105022993B discloses a five-line score playing method based on image recognition technology, including: image preprocessing and recognition step; five-line score removal step; note separation step; legato note segmentation step; note recognition step for each measure; and sounding step according to the corresponding frequency. By using the embodiment of the present invention, the melody of the score can be appreciated without understanding the score. At the same time, the method has low complexity, is easy to implement, and has great practical value.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems:

[0007] When identifying a staff image to be detected, there are complex decorative notes or overlapping notes, and the staff image to be detected does not exist in the database, resulting in a problem that it is impossible to determine whether the recognition result of the staff image to be detected is correct. Summary of the invention

[0008] The embodiments of the present application provide a method and device for staff recognition based on a sparrow search algorithm, thereby solving the problem of being unable to determine whether the recognition result of the staff image to be detected is correct when recognizing the staff image to be detected.

[0009] The embodiment of the present application provides a staff recognition method based on a sparrow search algorithm, comprising the following steps: firstly, merging a staff image to be detected with a preset staff image, identifying and removing the staff lines in the merged image; identifying each independent image in the image after the staff lines are removed, and dividing it into a first note image data set and a first symbol image data set; after quantizing the images of the first note image and the first symbol image into numerical values, matching the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image from a database through a sparrow search algorithm; obtaining the original audio corresponding to the preset staff image from the database, recorded as the second audio; and recording the preset staff image in the database. Suppose that some standard notes and standard symbols corresponding to the staff image generate a first audio, analyze the audio similarity evaluation value between the first audio and the second audio; compare the audio similarity evaluation value with the audio similarity evaluation value threshold, if the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the part of the standard notes and standard symbols corresponding to the staff image to be detected are output, otherwise, adjust the number of iterations and population size parameters of the sparrow search algorithm, and re-match the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0010] Furthermore, the staff image to be detected is merged with a preset staff image, and the steps of identifying and removing the staff lines in the merged image include: adjusting the notes and symbols in the staff image to be detected and the preset staff image to the same recognition level; merging the preset staff image with the staff image to be detected, recorded as: merging the image; analyzing the pixel points in the merged image to find the straight line segments that meet the characteristics of the staff lines; extracting the characteristic information of the staff lines, including position, length, and direction; and setting the pixel value at the position where the staff lines are located as the background color according to the extracted characteristic information.

[0011] Furthermore, each independent image in the image after removing the spectral lines is identified, divided into a first note image data set and a first symbol image data set, and their numbers are counted separately, including: converting the image after removing the spectral lines into a binary image; identifying independent images in the binary image as connected components; identifying and classifying connected components as a first note image data set and a first symbol image data set; and counting the numbers of the first note images and the first symbol images for subsequent calculations.

[0012] Furthermore, the steps of identifying and classifying connected components into a first note image data set and a first symbol image data set include: preparing in advance a template image containing standard staff notes and symbols; matching the connected components with the template image; and classifying the connected components into a first note image data set and a first symbol image data set based on the matching results.

[0013] Furthermore, after the images of the first note image and the first symbol image are quantized into numerical values, the step of matching the most similar note image and the most similar symbol image from the database through the sparrow search algorithm includes: quantizing the first note image and the first symbol image into numerical values; obtaining the quantized numerical values ​​of the standard note image and the standard symbol image in the database; and performing similarity matching between the image quantization numerical values ​​of the first note image and the first symbol image and the quantization numerical values ​​of the standard note image and the standard symbol image through the sparrow search algorithm.

[0014] Furthermore, the image quantization values ​​of the first note image and the first symbol image are matched with the quantization values ​​of the standard note image and the standard symbol image by a sparrow search algorithm, and the step of obtaining the most similar standard note image and standard symbol image includes: using the calculation formula of the first note image similarity evaluation value and the first symbol image similarity evaluation value as the objective function, searching in the quantitative feature space of the standard note image and the standard symbol image in the database, randomly generating an initial population, each individual in the population representing the quantization value of the first note image or the first symbol image; calculating the initial first note image similarity evaluation value and the first symbol image similarity evaluation value of the individuals in the population, sorting the individuals in the population from large to small according to the initial first note image similarity evaluation value and the first symbol image similarity evaluation value, and selecting a preset proportion of the individuals at the front of the sequence. The individual acts as a discoverer, and the remaining individuals act as joiners; the discoverer is responsible for extensive searching to find the potential maximum first note image similarity evaluation value and the first symbol image similarity evaluation value; the joiner is responsible for using the information provided by the discoverer to update and obtain a larger first note image similarity evaluation value and the first symbol image similarity evaluation value; traverse and update the positions of the individuals in the population, calculate the image similarity evaluation values ​​of the updated individuals in the population, and update the types of the individuals in the population according to the image similarity evaluation values; the traversal ends when the set number of traversals is reached, and the maximum first note image similarity evaluation value and the first symbol image similarity evaluation value are the quantized values ​​of the standard note image and the standard symbol image that are most similar to the first note image and the first symbol image, thereby obtaining the most similar standard note image and the standard symbol image to the first note image and the first symbol image.

[0015] Furthermore, the calculation formulas for the first note image similarity evaluation value and the first symbol image similarity evaluation value are as follows:

[0016]

[0017] In the formula, S i represents the similarity evaluation value of the first note image of the i-th image, λ represents the weight factor of the note image shape similarity, (1-λ) represents the weight factor of the note image position similarity, j i , k i , l i Respectively represent the weights of the note head, stem, and tail of the i-th first note image for the entire first note image note, x hi 、x mi 、x ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the i-th note in the first note image, y hi ,y mi ,y ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the standard note image matched with the i-th first note image in the database, x i ,y i represents the quantized value of the image position of the i-th note in the first note image and the quantized value of the standard note image position matched with the i-th first note image in the database, respectively. n represents the similarity evaluation value of the nth first symbol image, μ represents the weight of the symbol image shape similarity, (1-μ) represents the weight of the symbol image position similarity, m sn 、n sn The image of the nth symbol in the first symbol image and the quantized value of the shape of the standard symbol image in the database that matches the nth first symbol image are represented respectively, and m xn 、n xn They respectively represent the image of the nth symbol in the first symbol image and the quantized values ​​of the symbol positions of the standard symbol image in the database that matches the nth first symbol image. When i is the number of the note, i=1,2,3..., N1 represents the number of the note, n is the number of the symbol, n=1,2,3..., N2 represents the number of the symbol.

[0018] Furthermore, the first audio is generated by using some standard notes and standard symbols corresponding to the preset staff image, and the step of analyzing the audio similarity evaluation value between the first audio and the second audio includes: firstly, the first audio is generated by using some standard notes and standard symbols corresponding to the preset staff image, and the audio similarity evaluation value between the first audio and the second audio is converted into a digital audio signal; a waveform graph is generated from the digital audio signal, the period of the waveform is determined, and the frequency is calculated according to the period of the waveform; the digital audio signal is sampled to obtain the amplitude value of each sampling point; and the audio similarity evaluation value calculation formula is used to calculate the audio similarity evaluation value between the first audio and the second audio.

[0019] Furthermore, the calculation formula of the audio similarity evaluation value is:

[0020]

[0021] Where S represents the audio similarity evaluation value, η represents the weight factor of the time value similarity, (1-η represents the weight factor of the pitch and intensity similarity, and j k , k k Represent the weights of pitch and intensity at the kth sampling point, m lk 、m fk Respectively represent the pitch and intensity value of the kth sampling point in the first audio, m dj Indicates the duration of the jth note in the first audio, n lk 、n fk Represents the pitch and intensity value of the kth sampling point in the second audio, n dj represents the duration of the jth note in the second audio, φ k Represents the correction coefficient of the phase difference effect of the kth amplitude value, e represents the base of the natural logarithm, wherein the pitch is obtained by calculating the frequency of the sound waveform by measuring its period, the intensity is obtained based on the amplitude value of the sampling point, the duration of a single note is calculated based on the full note duration and the note type, j is the note number, j=1,2,3..., N1 represents the number of notes, k is the number of sampling points, k=1,2,3..., N, N represents the total number of sampling points.

[0022] The embodiment of the present application provides a staff recognition system based on a sparrow search algorithm, which includes: a staff line processing module, a data classification module, an image matching module, a second audio acquisition module, an audio similarity evaluation module, and a final image acquisition module; wherein the staff line processing module is used to merge the staff image to be detected with the preset staff image, identify the staff lines in the merged image and remove them; the data classification module is used to identify each independent image in the image after the staff lines are removed, divide it into a first note image data set and a first symbol image data set, and count their numbers respectively; the image matching module is used to quantize the images of the first note image and the first symbol image into numerical values, and then match the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image from the database through the sparrow search algorithm; the second audio The acquisition module is used to acquire the original audio corresponding to the preset staff image from the database, recorded as the second audio; the audio similarity evaluation module is used to generate the first audio from the part of the standard notes and standard symbols corresponding to the preset staff image, and analyze the audio similarity evaluation value between the first audio and the second audio; the final image acquisition module is used to compare the audio similarity evaluation value with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the part of the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and the population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0024] 1. By converting the screened images into audio and comparing them with the audio part of the preset staff image, the accuracy of image recognition is verified by referring to the audio similarity evaluation value, which further enhances the credibility of the results and effectively solves the problem in the prior art that it is impossible to determine whether the recognition result of the staff image to be detected is correct.

[0025] 2. The entire process is highly automated, from image merging, spectral line removal, object recognition and classification to feature quantification matching, and then to audio generation and comparison, which reduces manual intervention and improves processing efficiency and accuracy.

[0026] 3. By processing the image and optimizing the similarity evaluation value using the sparrow search algorithm, the notes and symbols in the image can be accurately identified and the most similar image can be matched. When the audio similarity evaluation value does not reach the audio similarity evaluation value threshold, the number of iterations and population size parameters of the algorithm can be adjusted to re-match the most similar standard note image and standard symbol image until the audio similarity evaluation value reaches the audio similarity evaluation value threshold, thereby ensuring the accuracy of the result. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of a five-line music score recognition method based on a sparrow search algorithm provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of the structure of a staff recognition system based on a sparrow search algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiment of the present application provides a staff recognition method and device based on a sparrow search algorithm, which solves the problem that when recognizing a staff image to be detected, due to the presence of complex decorative notes or overlapping notes, and the staff image to be detected does not exist in the database, it is impossible to determine whether the recognition result of the staff image to be detected is correct. The staff image to be detected is merged with a preset staff image, and the staff lines are removed to separate the staff image into a first note image and a first symbol image. The most similar standard note image and standard symbol image are matched by the sparrow search algorithm to generate a first audio. The original audio corresponding to the preset staff image is obtained from the database and recorded as the second audio. The audio similarity evaluation value of the first audio and the second audio is calculated, and the audio similarity evaluation value is compared with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and the population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image are re-matched until the audio similarity evaluation value reaches the audio similarity evaluation value threshold.

[0030] The technical solution in the embodiment of the present application solves the problem that when identifying a staff image to be detected, due to the presence of complex decorative notes or overlapping notes, and the staff image to be detected does not exist in the database, it is impossible to determine whether the recognition result of the staff image to be detected is correct. The overall idea is as follows: first, the staff image to be detected is merged with a preset staff image, and the staves in the merged image are identified and removed; each independent image in the image after the staves are removed is identified and divided into a first note image data set and a first symbol image data set; after quantizing the images of the first note image and the first symbol image into numerical values, the most similar standard note image and standard symbol corresponding one-to-one to the first note image and the first symbol image are matched from the database through a sparrow search algorithm. image; obtaining the original audio corresponding to the preset staff image from the database, recorded as the second audio; generating the first audio with part of the standard notes and standard symbols corresponding to the preset staff image, analyzing the audio similarity evaluation value between the first audio and the second audio; comparing the audio similarity evaluation value with the audio similarity evaluation value threshold, if the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the part of the standard notes and standard symbols corresponding to the staff image to be detected are output, otherwise, adjusting the number of iterations and population size parameters of the sparrow search algorithm, and re-matching the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0031] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0032] like Figure 1 , which is a flow chart of a staff recognition method based on a sparrow search algorithm provided in an embodiment of the present application. The method is applied to a staff recognition device based on a sparrow search algorithm. The method includes the following steps:

[0033] First, the staff image to be detected is merged with the preset staff image, and the staff lines in the merged image are identified and removed; each independent image in the image after the staff lines are removed is identified and divided into a first note image data set and a first symbol image data set; after the images of the first note image and the first symbol image are quantized into numerical values, the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are matched from the database through the sparrow search algorithm; the original audio corresponding to the preset staff image is obtained from the database and recorded as the second audio; some standard notes corresponding to the preset staff image are matched. and standard symbols to generate a first audio, analyze the audio similarity evaluation value between the first audio and the second audio; compare the audio similarity evaluation value with the audio similarity evaluation value threshold, if the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the standard notes and standard symbols corresponding to the staff image to be detected are output, otherwise, the number of iterations and population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

[0034] Furthermore, the staff image to be detected is merged with a preset staff image, and the steps of identifying and removing the staff lines in the merged image include: adjusting the notes and symbols in the staff image to be detected and the preset staff image to the same recognition level; merging the preset staff image with the staff image to be detected, denoted as a merged image; merging the preset staff image with the staff image to be detected includes a variety of methods, such as simple image splicing to keep the key notes and symbols at a uniform recognition level, or dividing the preset staff image into a single or multiple notes and symbols and interspersing them in the preset staff image, etc.; analyzing the pixel points in the merged image to find the straight line segments that meet the characteristics of the staff lines; extracting the characteristic information of the staff lines, including position, length, and direction; and setting the pixel value at the position of the staff lines as the background color according to the extracted characteristic information.

[0035] In this embodiment, the image merging can be a merging of the whole images, or it can be a merging after splitting the images into units of musical bars. The merging order can also be multiple. The preset staff image can be divided into multiple parts and inserted into different positions of the staff image to be detected, or the preset staff image can be placed in any orientation or position of the staff image to be detected, that is, in order to better integrate the preset staff image and the staff image to be detected, so that the preset staff image after recognition can be used as a judgment basis for verifying the recognition accuracy, the same recognition level indicates that the notes and symbols of the main parts of the two images belong to the same size level or resolution level. The size level can be pre-determined according to the size of the image and the size of the notes and symbols. For example, the size grades can be divided into 1, 2, 3, 4... grades. The first grade is the smallest size grade, specifically less than 1 mm, the second grade is 1-2 mm, and so on. This grading standard can make the image recognition process at a similar size to avoid recognition errors caused by different sizes. Similarly, consistent resolution levels mean that the resolution of the notes or symbols in the two pictures is adjusted to a unified level, for example, the clarity is adjusted to a consistent level, or the pixels are adjusted to a consistent level. In addition, if the clarity or pixels of the picture are poor, image restoration technology can be used in advance to repair or increase the clarity, fill in the missing images, and then adjust them. This operation can make the recognition complete even if the image to be processed is poor. When recognizing musical notation, the lines of the score often cross or overlap with most of the musical symbols, which destroys the unique features of the symbols and makes subsequent symbol recognition difficult. Deleting the lines can reduce recognition interference and make the features of the note symbols more prominent, which is conducive to improving the accuracy and efficiency of feature extraction. It can also simplify the processing flow of the musical notation image, reduce unnecessary calculations, and increase the recognition speed accordingly. In addition, the existence of the lines may cause the recognition system to mistakenly regard the lines as part of the musical symbols, resulting in recognition errors. Deleting the lines can enable the system to more accurately identify the musical symbols in the score, including notes, rests, dynamic marks, etc., which significantly improves the overall accuracy of musical notation recognition.

[0036] Furthermore, each independent image in the image after removing the spectral lines is identified, divided into a first note image data set and a first symbol image data set, and their numbers are counted separately, including: converting the image after removing the spectral lines into a binary image so that the pixels in the image have only two possibilities, 0 and 255; identifying independent images in the binary image as connected components; identifying and classifying connected components as a first note image data set and a first symbol image data set; and counting the number of first note images and first symbol images for subsequent calculations.

[0037] In this embodiment, independent images are segmented based on whether the first note image or the first symbol image is connected. For example, triplets, quadruplets and other notes will be treated as a whole as one note data. When the image after removing the spectrum lines is segmented into independent images, it is first necessary to traverse each pixel point of the binary image, and assign the same label to the pixels belonging to the same connected component based on the value of the pixel point and the relationship between the adjacent pixels.

[0038] Furthermore, the steps of identifying and classifying connected components into a first note image data set and a first symbol image data set include: preparing in advance a template image containing standard staff notes and symbols; matching the connected components with the template image; and classifying the connected components into a first note image data set and a first symbol image data set based on the matching results.

[0039] In this embodiment, when matching the connected component with the template image, possible note and symbol areas are screened out based on the size, shape and other features of the connected component, the morphological features of the connected component and the template image are extracted, and a suitable data set is selected for matching. Based on the matched data set, the connected component is divided into a first note image data set and a first symbol image data set. In addition, when the image contains multiple notes and symbols and there are specific spatial relationships or layout rules between them, contextual relevance is used to improve the accuracy of classification.

[0040] Furthermore, after the images of the first note image and the first symbol image are quantized into numerical values, the step of matching the most similar note image and the most similar symbol image from the database through the sparrow search algorithm includes: first quantizing the first note image and the first symbol image into numerical values; obtaining the quantized numerical values ​​of the standard note image and the standard symbol image in the database; and performing similarity matching between the image quantization numerical values ​​of the first note image and the first symbol image and the quantization numerical values ​​of the standard note image and the standard symbol image through the sparrow search algorithm.

[0041] In this embodiment, this method has important benefits and practical application value in music image processing and digital archive management: by calculating the similarity with existing stave images, images similar to specific staves can be retrieved more accurately from a large number of databases, which is very useful for music scholars, educators, and database managers, and can quickly locate the required information; automated image similarity evaluation and retrieval can greatly improve work efficiency, especially when processing large-scale databases, this method reduces the need for manual retrieval, saving a lot of time and labor resources; at the same time, allowing users to choose from similar images, providing greater flexibility and personalized options, users can choose the most desired image according to their specific needs, enhancing user experience experience and satisfaction; in music archives and digital library projects, it is crucial to be able to effectively organize and retrieve image materials. The method can help institutions better manage their image materials and improve the accessibility and usability of data; scholars and students can also use the method to find specific five-line scores to support research and teaching activities related to musicology, history or art, such as studying the historical evolution of different versions of five-line scores, or analyzing the style of a composer's works; by providing multiple images similar to specific five-line scores, it can also inspire users to have new insights and creative thinking about different musical works; in addition, the implementation of image similarity retrieval technology can promote the development of related technologies, such as image processing algorithms, machine learning models and user interface design, and further improve the overall technical level and application scope.

[0042] Furthermore, the image quantization values ​​of the first note image and the first symbol image are matched with the quantization values ​​of the standard note image and the standard symbol image by a sparrow search algorithm, and the step of obtaining the most similar standard note image and standard symbol image includes: using the calculation formula of the first note image similarity evaluation value and the first symbol image similarity evaluation value as the objective function, searching in the quantitative feature space of the standard note image and the standard symbol image in the database, randomly generating an initial population, each individual in the population representing the quantization value of the first note image or the first symbol image; calculating the initial first note image similarity evaluation value and the first symbol image similarity evaluation value of the individuals in the population, sorting the individuals in the population from large to small according to the initial first note image similarity evaluation value and the first symbol image similarity evaluation value, and selecting a preset proportion of the individuals at the front of the sequence. The individual acts as a discoverer, and the remaining individuals act as joiners; the discoverer is responsible for extensive searching to find the potential maximum first note image similarity evaluation value and the first symbol image similarity evaluation value; the joiner is responsible for using the information provided by the discoverer to update and obtain a larger first note image similarity evaluation value and the first symbol image similarity evaluation value; traverse and update the positions of the individuals in the population, calculate the image similarity evaluation values ​​of the updated individuals in the population, and update the types of the individuals in the population according to the image similarity evaluation values; the traversal ends when the set number of traversals is reached, and the maximum first note image similarity evaluation value and the first symbol image similarity evaluation value are the quantized values ​​of the standard note image and the standard symbol image that are most similar to the first note image and the first symbol image, thereby obtaining the most similar standard note image and the standard symbol image to the first note image and the first symbol image.

[0043] In this embodiment, the sparrow search algorithm optimizes and solves the problem by simulating the process of sparrows looking for food in nature. Through the behaviors of foraging, clustering, jumping, and escaping, it is efficient and accurate in solving the optimization problem. The sparrow search algorithm has a simple structure, fast convergence speed, and strong global search capability, which is conducive to quickly optimizing the similarity of the five-line score image.

[0044] Furthermore, the calculation formulas for the first note image similarity evaluation value and the first symbol image similarity evaluation value are as follows:

[0045]

[0046] In the formula, S i represents the similarity evaluation value of the first note image of the i-th image, λ represents the weight factor of the note image shape similarity, (1-λ) represents the weight factor of the note image position similarity, j i , k i , l i Respectively represent the weights of the note head, stem, and tail of the i-th first note image for the entire first note image note, x hi、x mi 、x ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the i-th note in the first note image, y hi ,y mi ,y ei Respectively represent the quantized values ​​of the note head, stem, and tail images of the standard note image matched with the i-th first note image in the database, x i ,y i represents the quantized value of the image position of the i-th note in the first note image and the quantized value of the standard note image position matched with the i-th first note image in the database, respectively. n represents the similarity evaluation value of the nth first symbol image, μ represents the weight of the symbol image shape similarity, (1-μ) represents the weight of the symbol image position similarity, m sn 、n sn The image of the nth symbol in the first symbol image and the quantized value of the shape of the standard symbol image in the database that matches the nth first symbol image are represented respectively, and m xn 、n xn represents the image of the nth symbol in the first symbol image and the quantized value of the symbol position of the standard symbol image matched with the nth first symbol image in the database, i is the number of the note, i=1,2,3..., N1 represents the number of the note, n is the number of the symbol, n=1,2,3..., N2 represents the number of the symbol, S 总 Represents the total similarity evaluation value.

[0047] The total similarity evaluation value formula calculates the similarity of the total merged image and the composite image of the recognized standard musical note image and the standard symbol image. If the staff image to be detected is separated from the preset staff image and the most similar standard musical note image and standard symbol image that are matched one-to-one with the first musical note image and the first symbol image, and the most similar standard musical note image and standard symbol image that are matched one-to-one with the first musical note image and the first symbol image are combined into a second preset staff image, S 总 The total similarity evaluation value between the second preset stave image and the preset stave image is calculated by the formula, which can indirectly reflect the accuracy of the recognition result of the stave image to be detected.

[0048] In this embodiment, the images and positions of notes and symbols are quantized, which involves the comparison of image features and the accurate representation of spatial positions, wherein the image quantization method includes: first determining the range of quantization values, which determines how many different levels the images of notes and symbols will be divided into, traversing each note image and symbol image, and splitting the note image and symbol image into feature shapes with a fixed number according to the size and shape of the allocated area of ​​the pixels of each note image and symbol image, and assigning quantization values ​​to the shape features of each note image and symbol image, one quantization value represents a fixed feature shape, and the closer the quantization values ​​are, the more similar the feature shapes are; combining the quantization values ​​into a feature vector, which can represent the quantized image of a single note image or symbol image; position quantization generally refers to the accurate representation of the spatial coordinates of an object or feature point in an image, and in this embodiment, the position quantization of the note image and symbol image refers to the position of each note image and symbol image relative to the original position of the image. The coordinate value of the position of the five-line staff line is added again before the staff line is removed or after the operation step of splitting into each independent image, so as to judge each note and the position on the corresponding staff line. For the notes on the staff line, a coordinate value is set for each line and space in the five-line staff line, and the coordinate value is the position quantization value of the note and symbol, such as the coordinate value of the note on the third line is 3, the coordinate value of the note on the third space is 3.5, etc.; when the note exceeds the range of the five-line staff line, the position is added with a line, and the coordinate value is set for each added line and line space, such as the coordinate value of the first added line above the staff line is 6, and the coordinate value of the first added line below the staff line is 0; various symbols such as clef, key signature, time signature, etc. often appear in the five-line staff, and the positions of these symbols are usually fixed, such as the treble clef is located above the second line, and the coordinate value of the position is calculated according to the position; for other special symbols such as dots, slurs, etc., the value is set according to their position on the note, so as to realize the quantification of the position of the note image and the symbol image.

[0049] Furthermore, the first audio is generated by using some standard notes and standard symbols corresponding to the preset staff image, and the step of analyzing the audio similarity evaluation value between the first audio and the second audio includes: firstly, the first audio is generated by using some standard notes and standard symbols corresponding to the preset staff image, and the audio similarity evaluation value is converted into a digital audio signal; the digital audio signal is displayed as a waveform graph, and specifically, the corresponding audio waveform can be generated by analog audio software, or it can be input into a device that can automatically analyze sound signals to obtain the corresponding waveform, determine the period of the waveform, and calculate the frequency according to the period of the waveform; the digital audio signal is sampled to obtain the amplitude value of each sampling point; and the audio similarity evaluation value calculation formula is used to calculate the audio similarity evaluation value between the first audio and the second audio.

[0050] In this embodiment, a method for evaluating the similarity between the generated audio and the audio portion corresponding to the stave image by digital processing is described, wherein the formula for calculating the frequency based on the period of the waveform is:

[0051] Where f is the frequency of the waveform and T is the period of the waveform.

[0052] Furthermore, the calculation formula of the audio similarity evaluation value is:

[0053]

[0054] Where S represents the audio similarity evaluation value, η represents the weight factor of the time value similarity, (1-η represents the weight factor of the pitch and intensity similarity, and j k , k k Represent the weights of pitch and intensity at the kth sampling point, m lk 、m fk Respectively represent the pitch and intensity value of the kth sampling point in the first audio, m dj Indicates the duration of the jth note in the first audio, n lk 、n fk Represents the pitch and intensity value of the kth sampling point in the second audio, n dj represents the duration of the jth note in the second audio, φ k Represents the correction coefficient of the phase difference effect of the kth amplitude value, e represents the base of the natural logarithm, wherein the pitch is obtained by calculating the frequency of the sound waveform by measuring its period, the intensity is obtained based on the amplitude value of the sampling point, the duration of a single note is calculated based on the full note duration and the note type, j is the note number, j=1,2,3..., N1 represents the number of notes, k is the number of sampling points, k=1,2,3..., N, N represents the total number of sampling points.

[0055] In this embodiment, the audio similarity evaluation value calculation formula is mainly used to quantify the similarity between two or more audio signals, and can be calculated based on different features of the audio signals to obtain a similarity score, which directly reflects the similarity or difference between the audio signals. In applications such as audio search, recommendation, and copyright detection, the audio similarity evaluation value calculation formula can significantly improve processing efficiency, quickly find similar audio files or determine the copyright of the audio; in applications such as music recommendation and noise elimination, the application of the audio similarity evaluation value calculation formula can provide users with a more personalized and high-quality audio experience; with the continuous development of technologies such as deep learning, the audio similarity evaluation value calculation formula is also constantly innovating and optimizing, bringing more technological breakthroughs and application innovations to the field of audio processing.

[0056] Furthermore, the system includes: a spectrum processing module, a data classification module, an image matching module, a second audio acquisition module, an audio similarity evaluation module, and a final image acquisition module; wherein the spectrum processing module is used to merge the five-line spectrum image to be detected with the preset five-line spectrum image, identify the spectrum in the merged image and remove it; the data classification module is used to identify each independent image in the image after the spectrum is removed, divide it into a first note image data set and a first symbol image data set, and count their numbers respectively; the image matching module is used to quantize the images of the first note image and the first symbol image into numerical values, and then match the most similar standard note image and standard symbol image that correspond one-to-one to the first note image and the first symbol image from the database through a sparrow search algorithm; the second audio acquisition module is used to obtain the original audio corresponding to the preset five-line spectrum image from the database, recorded as a second audio; the audio similarity evaluation module is used to generate a first audio from some standard notes and standard symbols corresponding to a preset staff image, and analyze the audio similarity evaluation value between the first audio and the second audio; the final image acquisition module is used to compare the audio similarity evaluation value with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the standard notes and standard symbols corresponding to the staff image to be detected are output; otherwise, by adjusting the number of iterations and the population size parameter of the sparrow search algorithm, the standard notes and standard symbols in the database are added, and the most similar standard note images and standard symbol images corresponding to the first note image and the first symbol image are re-matched in the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, wherein adjusting the population size parameter is to adjust the number of discoverers and joiners.

[0057] like Figure 2 As shown, it is a structural schematic diagram of the staff notation recognition system based on the sparrow search algorithm provided in the embodiment of the present application. The staff notation recognition system based on the sparrow search algorithm provided in the embodiment of the present application includes: a staff line processing module, a data classification module, an image matching module, a second audio acquisition module, an audio similarity evaluation module, and a final image acquisition module.

[0058] In summary, the embodiment of the present application merges the to-be-detected and preset five-line staff images, and then removes the staff lines in the merged image; classifies and counts the first note image and the first symbol image in the image after the staff lines are removed; after quantifying the first note image and the first symbol image, the most similar standard note image and standard symbol image are matched through the sparrow search algorithm to generate the first audio, and the original audio corresponding to the preset five-line staff image is obtained from the database, recorded as the second audio, and the audio similarity evaluation value of the first audio and the second audio is calculated. When the audio similarity evaluation value reaches the audio similarity evaluation value threshold, the current image is selected as the final result. If the audio similarity evaluation value threshold is not reached, the parameters of the sparrow search algorithm are adjusted to re-match the most similar standard note and standard symbol image.

[0059] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0063] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0064] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A five-line score recognition method based on a sparrow search algorithm, characterized in that: The method comprises the following steps: Merge the stave image to be detected with the preset stave image, identify and remove the stave lines in the merged image; identifying each independent image in the image after the staves are removed, and dividing it into a first musical note image dataset and a first symbol image dataset; After quantizing the first musical note image and the first symbol image into numerical values, the most similar standard musical note image and standard symbol image corresponding to the first musical note image and the first symbol image are matched from a database by a sparrow search algorithm; Obtaining the original audio corresponding to the preset stave image from the database, recorded as the second audio; Generate a first audio using part of the standard notes and standard symbols corresponding to the preset stave image, and analyze an audio similarity evaluation value between the first audio and the second audio; The audio similarity evaluation value is compared with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed, and the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and the population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched from the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold. After quantizing the first musical note image and the first symbol image into numerical values, the step of matching the most similar musical note image and the most similar symbol image from the database by using a sparrow search algorithm comprises: quantizing the first musical note image and the first symbol image into numerical values; Obtain quantized values ​​of standard musical note images and standard symbol images in the database; The image quantization values ​​of the first musical note image and the first symbol image are matched with the quantization values ​​of the standard musical note image and the standard symbol image by using a sparrow search algorithm to obtain the most similar standard musical note image and standard symbol image; The step of performing similarity matching between the image quantization values ​​of the first note image and the first symbol image and the quantization values ​​of the standard note image and the standard symbol image by using the sparrow search algorithm to obtain the most similar standard note image and standard symbol image comprises: The calculation formula of the similarity evaluation value of the first note image and the first symbol image is used as the objective function, the search range is the quantitative feature space of the standard note image and the standard symbol image in the database, and an initial population is randomly generated, each individual in the population represents the quantitative value of the first note image or the first symbol image; Calculate the initial first note image similarity evaluation value and the first symbol image similarity evaluation value of the population individuals, sort the population individuals from large to small according to the initial first note image similarity evaluation value and the first symbol image similarity evaluation value, select a preset proportion of individuals at the front of the sequence as discoverers, and the remaining individuals as joiners; The finder is responsible for extensive searching to find the potential maximum first note image similarity evaluation value and the first symbol image similarity evaluation value; The joiner is responsible for updating the information provided by the finder to obtain a larger first note image similarity evaluation value and first symbol image similarity evaluation value; Traverse and update the positions of the individuals in the population, calculate the image similarity evaluation values ​​of the updated individuals in the population, and update the types of the individuals in the population according to the image similarity evaluation values; The traversal ends when the set number of traversals is reached. The maximum first note image similarity evaluation value and the first symbol image similarity evaluation value are the quantized values ​​of the standard note image and the standard symbol image that are most similar to the first note image and the first symbol image, thereby obtaining the most similar standard note image and the standard symbol image to the first note image and the first symbol image.

2. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: The step of merging the to-be-detected stave image with the preset stave image, and identifying and removing the staves in the merged image comprises: Adjusting the notes and symbols in the to-be-detected stave image and the preset stave image to the same recognition level; Merge the preset stave image with the stave image to be detected, and record it as: merged image; Analyze the pixels in the merged image and find the straight line segments that meet the spectral line characteristics; Extract characteristic information of spectral lines, including position, length, and direction; According to the extracted feature information, the pixel value at the location of the spectrum line is set as the background color.

3. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: The steps of identifying each independent image in the image after the spectrum is removed, and dividing it into a first musical note image data set and a first symbol image data set, specifically include: Convert the image after removing the spectral lines into a binary image; Identify independent images in a binary image as connected components; identifying and classifying connected components as a first musical note image dataset and a first symbol image dataset; The number of the first musical note images and the first symbol images is counted.

4. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 3, characterized in that: The steps of identifying and classifying the connected components as the first musical note image dataset and the first symbol image dataset include: Prepare template images containing standard stave notes and symbols in advance; Perform matching operation on the connected components and the template image; According to the matching result, the connected components are classified into a first musical note image dataset and a first symbol image dataset.

5. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: The calculation formula of the first note image similarity evaluation value and the first symbol image similarity evaluation value is: ； ； In the formula, represents the similarity evaluation value of the i-th first note image, represents the weight factor of the note image shape similarity, (1-λ) represents the weight factor of the note image position similarity, , , Respectively represent the weights of the note head, stem, and tail of the i-th first note image for the entire first note image note, , , Respectively represent the quantized values ​​of the note head, stem, and tail images of the i-th note in the first note image, , , They respectively represent the quantized values ​​of the note head, stem, and tail images of the standard note image matched with the i-th first note image in the database, , represent the quantized value of the image position of the i-th note in the first note image and the quantized value of the standard note image position matched with the i-th first note image in the database, respectively. represents the nth first symbol image similarity evaluation value, represents the weight of the symbol image shape similarity, (1-μ) represents the weight of the symbol image position similarity, , represents the image of the nth symbol in the first symbol image and the quantized value of the shape of the standard symbol image in the database that matches the nth first symbol image, , Respectively represent the image of the nth symbol in the first symbol image and the quantized value of the symbol position of the standard symbol image in the database that matches the nth first symbol image, when i is the number of the note, i=1,2,3..., , Represents the number of notes, n is the symbol number, n=1,2,3..., , Represents the number of symbols.

6. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 1, characterized in that: The step of generating a first audio with part of the standard notes and standard symbols corresponding to the preset stave image, and analyzing the audio similarity evaluation value between the first audio and the second audio comprises: Firstly, some standard notes and standard symbols corresponding to the preset stave image are used to generate a first audio, and the first audio is converted into a digital audio signal; Generate a waveform diagram of the digital audio signal, determine the period of the waveform, and calculate the frequency based on the period of the waveform; Sampling the digital audio signal to obtain the amplitude value of each sampling point; The audio similarity evaluation value calculation formula is used to calculate the audio similarity evaluation value of the first audio and the second audio.

7. The method for identifying five-line musical notations based on the sparrow search algorithm as claimed in claim 6, characterized in that: The calculation formula of the audio similarity evaluation value is: ； In the formula, S represents the audio similarity evaluation value, The weight factor representing the similarity of the time value, The weighting factors representing the similarity of pitch and intensity, , Respectively represent the weights of pitch and intensity in the kth sampling point, , Respectively represent the pitch and intensity value of the kth sampling point in the first audio, represents the duration of the jth note in the first audio, , represents the pitch and intensity value of the kth sampling point in the second audio, represents the duration of the jth note in the second audio, represents the correction coefficient of the phase difference of the kth amplitude value, e represents the base of the natural logarithm, j is the number of the note, j=1,2,3..., , Represents the number of notes, k is the number of sampling points, k=1,2,3...,N, N represents the total number of sampling points.

8. A staff recognition system based on a Mahjong search algorithm, using a staff recognition method based on a Mahjong search algorithm as claimed in any one of claims 1 to 7, characterized in that: include: Spectrum line processing module, data classification module, image matching module, second audio acquisition module, audio similarity evaluation module, final image acquisition module; The spectrum line processing module is used to merge the to-be-detected five-line spectrum image with the preset five-line spectrum image, identify the spectrum lines in the merged image and remove them; The data classification module is used to identify each independent image in the image after the spectrum is removed, divide it into a first musical note image data set and a first symbol image data set, and count their numbers respectively; The image matching module is used to quantize the first musical note image and the first symbol image into numerical values, and then match the most similar standard musical note image and standard symbol image that correspond one-to-one to the first musical note image and the first symbol image from a database through a sparrow search algorithm; The second audio acquisition module is used to acquire the original audio corresponding to the preset stave image from the database, which is recorded as the second audio; The audio similarity evaluation module is used to generate a first audio from part of the standard notes and standard symbols corresponding to the preset stave image, and analyze the audio similarity evaluation value between the first audio and the second audio; The final image acquisition module is used to compare the audio similarity evaluation value with the audio similarity evaluation value threshold. If the audio similarity evaluation value is greater than the audio similarity evaluation value threshold, the recognition is completed and the standard notes and standard symbols corresponding to the staff image to be detected are output. Otherwise, the number of iterations and population size parameters of the sparrow search algorithm are adjusted, and the most similar standard note image and standard symbol image corresponding to the first note image and the first symbol image are re-matched from the database until the audio similarity evaluation value is greater than the audio similarity evaluation value threshold.

Images