Music score recognition method and device, electronic equipment and storage medium

Abstract

The application discloses a music score recognition method and device, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first music score image; the first music score image is cropped to obtain M first image blocks, and each first image block in the M first image blocks contains at least one note; M is a positive integer; a first target candidate box of each note in each first image block is acquired; a second image block corresponding to each note is cropped in the first music score image with the position corresponding to the first candidate box of each note in the first music score image as the center; a second target candidate box of each note in the second image block corresponding to each note is acquired; and the first music score image is recognized according to the second candidate box of each note; wherein the resolution of the second image block is greater than the resolution of the first music score image block.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a music score recognition method, device, electronic device, and storage medium. Background Technology

[0002] Currently, users can use electronic devices to recognize and process the first musical score image to obtain musical score information related to the score (such as the score name or the composer).

[0003] In related technologies, electronic devices input a first musical score image into a multi-target detection network model to detect the color and texture features of the notes in the first musical score image, and then determine the musical score information based on the color and texture features of the notes.

[0004] However, since musical notes occupy a small portion of the first musical score image, they may appear as black dots. This can cause electronic devices to fail to accurately identify the color and texture features of the musical notes, resulting in poor accuracy in recognizing musical scores. Summary of the Invention

[0005] The purpose of this application is to provide a music score recognition method, device, storage medium, and electronic device that can solve the problem of poor accuracy in music score recognition by electronic devices.

[0006] In a first aspect, embodiments of this application provide a music score recognition method, which includes: acquiring a first music score image; cropping the first music score image to obtain M first image blocks, each of the M first image blocks containing at least one musical note; M being a positive integer; acquiring a first target candidate box for each musical note in each first image block, the first candidate box for each musical note corresponding to the position information of each musical note in each first music score image block; cropping a second image block corresponding to each musical note from the first music score image, centered on the position corresponding to the first candidate box for each musical note on the first music score image; acquiring a second target candidate box for each musical note in the second image block corresponding to each musical note, the second candidate box for each musical note corresponding to the position information of each musical note in the second image block; and recognizing the first music score image based on the second candidate box for each musical note; wherein the resolution of the second image block is greater than the resolution of the first music score image block.

[0007] Secondly, embodiments of this application provide a music score recognition device, which includes an acquisition module, a cropping module, and a processing module. The acquisition module is used to acquire a first music score image. The cropping module is used to crop the first music score image to obtain M first image blocks, each of the M first image blocks containing at least one note; M is a positive integer. The acquisition module is further used to acquire a first target candidate box for each note in each first image block, the first candidate box for each note corresponding to the position information of each note in each first music score image block. The cropping module is further used to crop a second image block corresponding to each note in the first music score image, centered on the position corresponding to the first candidate box of each note on the first music score image. The acquisition module is further used to acquire a second target candidate box for each note in the second image block corresponding to each note, the second candidate box for each note corresponding to the position information of each note in the second image block. The processing module is used to recognize the first music score image based on the second candidate box of each note; wherein the resolution of the second image block is greater than the resolution of the first image block.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, the electronic device can crop a first musical score image to obtain M image blocks, and based on the M first image blocks, obtain a first target candidate box for each note in each of the M first image blocks, and crop out a second image block corresponding to each note in the first musical score image with the position corresponding to the first target candidate box of each note as the center; thereby obtaining a second target candidate box for each note in the second image block corresponding to each note, and recognizing the first musical score image based on the second candidate box of each note, wherein the resolution of the second image block is greater than that of the first image block. In this scheme, the electronic device first crops the first musical score image to obtain a first target candidate box for each note, thereby initially identifying the first position information of the notes in the first musical score image. Then, based on the first target candidate box, the first musical score image is cropped a second time to obtain a second image block with higher resolution. Based on the second image block with higher resolution, a second target candidate box with higher accuracy is identified for the notes in the first musical score image. The position information of each note in the second image block in the first musical score image is determined by the second target candidate box. Then, the first musical score image is identified using the second candidate box for each note. Through the second cropping, the electronic device can further accurately identify the position of the notes in the first musical score image. Thus, the musical score can be accurately identified based on the position of the notes in the first musical score image, thereby improving the accuracy of the electronic device in recognizing musical scores. Attached Figure Description

[0013] Figure 1 This is one of the flowcharts of a music score recognition method provided in the embodiments of this application;

[0014] Figure 2 This is one of the schematic diagrams illustrating an example of a music score recognition method provided in this application embodiment;

[0015] Figure 3 This is a second schematic diagram of a music score recognition method provided in an embodiment of this application;

[0016] Figure 4 This is a third schematic diagram of a music score recognition method provided in an embodiment of this application;

[0017] Figure 5 This is a fourth schematic diagram of a music score recognition method provided in the embodiments of this application;

[0018] Figure 6 This is a schematic diagram of the structure of a music score recognition device provided in an embodiment of this application;

[0019] Figure 7 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0020] Figure 8 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] The music score recognition method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0024] Currently, with the development of electronic devices, their functions are becoming increasingly diverse. For example, users can use electronic devices to recognize sheet music to obtain information such as the name or author of the music. In related technologies, electronic devices can use multi-object detection network models (such as SSD, YOLO, or CenterNet) to recognize sheet music and obtain this information. Specifically, users can input high-resolution sheet music images into the multi-object detection network model. To improve the model's processing speed, the model converts the high-resolution image into a low-resolution image (e.g., 256×256 or 512×512). Since musical notes occupy a small portion of the sheet music image, they may appear as black dots in the low-resolution image, making it difficult for the electronic device to accurately identify them. Furthermore, the lack of color and texture information makes it difficult for the multi-object detection network model to extract the deep neural network features needed for classification and recognition. Consequently, the accuracy of sheet music recognition by electronic devices is poor.

[0025] In this embodiment, the electronic device can crop the first musical score image to obtain M image blocks, and based on the M first image blocks, obtain a first target candidate box for each note in each of the M first image blocks, and crop out a second image block corresponding to each note in the first musical score image with the position corresponding to the first target candidate box of each note as the center; obtain a second target candidate box for each note in the second image block corresponding to each note, and thus identify the first musical score image based on the second candidate box of each note, wherein the resolution of the second image block is greater than that of the first image block. In this scheme, the electronic device first crops the first musical score image to obtain a first target candidate box for each note, thereby initially identifying the first position information of the notes in the first musical score image. Then, based on the first target candidate box, the first musical score image is cropped a second time to obtain a second image block with higher resolution. Based on the second image block with higher resolution, a second target candidate box with higher accuracy is identified for the notes in the first musical score image. The position information of each note in the second image block in the first musical score image is determined by the second target candidate box. Then, the first musical score image is identified using the second candidate box for each note. Through the second cropping, the electronic device can further accurately identify the position of the notes in the first musical score image. Thus, the musical score can be accurately identified based on the position of the notes in the first musical score image, thereby improving the accuracy of the electronic device in recognizing musical scores.

[0026] The execution entity of the music score recognition method provided in this application embodiment can be a music score recognition device, which can be an electronic device or a functional module in an electronic device. The following uses an electronic device as an example to illustrate the technical solution provided in this application embodiment.

[0027] This application provides a method for music score recognition. Figure 1 A flowchart of a music score recognition method provided in an embodiment of this application is shown. Figure 1 As shown, the music score recognition method provided in this application embodiment may include the following steps 201 to 206.

[0028] Step 201: The electronic device acquires the first musical score image.

[0029] In this embodiment of the application, the electronic device can identify the first musical score image to determine the position information of the notes in the first musical score image, and then obtain the musical score information of the first musical score image based on the position information of the notes.

[0030] Specifically, the electronic device can identify the position information of the noteheads in the first musical score image, and thus obtain the musical score information of the first musical score image based on the position information of the noteheads.

[0031] Optionally, in this embodiment of the application, the first musical score image can be a musical score image selected by the user; or, the first musical score image can be a musical score image captured by the electronic device through a camera.

[0032] Specifically, the sheet music image selected by the user can be one selected by the user from a target application (such as a photo album application) or one downloaded by the user through a browser application on an electronic device.

[0033] Optionally, in this embodiment, the camera may include at least one of the following: a telephoto camera, a macro camera, a wide-angle camera, and an ultra-wide-angle camera. The specific type can be determined based on actual usage requirements, and this embodiment does not impose any limitations.

[0034] Optionally, in the embodiments of this application, the first musical score image can be any of the following: a staff notation image, a guitar tablature image, or a guqin (a traditional Chinese stringed instrument) score image.

[0035] Optionally, in the embodiments of this application, the above-mentioned musical score information may include any of the following: musical score name, musical score author, and musical score pitch.

[0036] Optionally, in this embodiment of the application, the user can input the first musical score image into the image recognition control by making a first input, so that the electronic device can recognize the first musical score image through the recognition module to obtain the musical score information of the first musical score image.

[0037] Optionally, in this embodiment, the first input can be a user's click input, long press input, swipe input, or preset trajectory input on the first musical score image. Specifically, it can be determined according to actual usage needs, and this embodiment does not impose any limitations.

[0038] Optionally, in this embodiment of the application, after acquiring the first musical score image, the electronic device can obtain the optimal segmentation threshold (e.g., 140) of the pixel values ​​in the first musical score image through a target threshold segmentation algorithm. Then, the electronic device can perform binarization processing on the first musical score image to obtain the pixel values ​​of the pixels in the binarized first musical score image (hereinafter referred to as the first pixel values). The pixel values ​​of the pixels whose first pixel values ​​are greater than or equal to the optimal segmentation threshold are assigned the value 255 (i.e., white), and the pixel values ​​of the pixels whose first pixel values ​​are less than the optimal segmentation threshold are assigned the value 0 (i.e., black).

[0039] Specifically, the target threshold segmentation algorithm mentioned above can be the Otsu method.

[0040] Optionally, in this embodiment of the application, after the electronic device obtains the first musical score image (hereinafter referred to as the fifth musical score image) after binarization, the electronic device can detect horizontal lines in the fifth musical score image through a target line detection algorithm and retain horizontal lines of a preset length (e.g., more than half the image width of the fifth musical score image).

[0041] Specifically, the target line detection algorithm mentioned above can be the Hough line detection algorithm.

[0042] Optionally, in this embodiment of the application, if the line detected by the electronic device through the Hough line detection algorithm is not horizontal, the electronic device can convert at least one tilted line in the fifth musical score image into at least one horizontal line by rotating the tilt angle.

[0043] For example, such as Figure 2 As shown in (A), after the electronic device obtains at least one slanted straight line from the fifth musical score image, as... Figure 2 As shown in (B), the electronic device can obtain at least one tilt angle between at least one first straight line and a horizontal straight line using the Hough line detection algorithm, remove the maximum and minimum values ​​of the at least one tilt angle, and obtain the average value of the remaining tilt angles. Then, the endpoints of at least one first straight line are input into the rotation formula (Formula 1 below) to obtain the endpoints of the horizontal straight line. The at least one tilted straight line is then rotated according to the endpoints of the horizontal straight line to obtain the horizontal straight line. The specific Formula 1 is as follows:

[0044] (x1,y1)=(x,y)*[1,1-cos(alpha)] T =(x,y*(1-cos(alpha))) (Formula 1)

[0045] Where x1, y1 are the endpoints of the horizontal line, x, y are the endpoints of the inclined line, and alpha is the average value of the remaining inclination angle.

[0046] It should be noted that the above straight lines include horizontal and vertical lines, and the above endpoints include left and right endpoints. The electronic device can rotate the inclined straight lines according to the endpoints of the horizontal straight lines to obtain horizontal lines and vertical lines.

[0047] Optionally, in this embodiment of the application, after obtaining the horizontal lines and vertical lines, the electronic device can calculate the intersection points between each of the at least one horizontal line and each of the at least one vertical line, randomly select the intersection points between two horizontal lines and two vertical lines, calculate the perspective correction matrix, and then perform perspective correction on the fifth musical score image after rotation correction using the perspective correction matrix.

[0048] For example, such as Figure 2As shown in (C), after obtaining the intersection point between the two horizontal lines and the vertical line, the electronic device can input the intersection point between the two horizontal lines and the vertical line into the cvGetPerspectiveTransform function in the OpenCV (i.e., open source computer vision algorithm library) library to obtain the correction matrix. Then, the electronic device can call the perspective transformation function in OpenCV (such as cvWarpPerspective) to obtain the perspective-corrected fifth musical score image.

[0049] In this embodiment, the electronic device performs perspective transformation on the first musical score image, eliminating the need for the user to repeatedly correct the position of the musical score image captured by the electronic device before the electronic device can accurately identify the musical score information. This simplifies user operation and improves the flexibility of the electronic device in recognizing musical score images.

[0050] Optionally, in this embodiment of the application, after the electronic device obtains the first musical score image, the electronic device can convert the first musical score image from an RGB color channel to a single-channel grayscale image through a first method.

[0051] Optionally, in the embodiments of this application, the first method described above can be any of the following: average method, weighted average method, maximum value method, or maximum-minimum value average method.

[0052] It is understandable that since the notes in a musical score image are generally black and the background is generally light-colored, the color information of the musical score image does not provide substantial help in recognizing the notes. Therefore, in this embodiment of the application, converting the first musical score image from the RGB color channel to a single-channel grayscale image can reduce the computational load of the electronic device, thereby improving the computational speed of the electronic device in recognizing the musical score image.

[0053] Optionally, in the embodiments of this application, step 201 above can be specifically implemented by step 201a below.

[0054] Step 201a: The electronic device constructs a multi-layered musical score image based on the original musical score image.

[0055] In this embodiment of the application, the above-mentioned multi-layered musical score image is the first musical score image; each layer of the multi-layered musical score image has a different image size.

[0056] In this application, the electronic device can obtain at least one second musical score image based on the original musical score image by scaling it proportionally. Then, the electronic device can use the original musical score image as the first layer of a multi-layer musical score image and superimpose at least one second musical score image on top of the original musical score image to obtain a multi-layer musical score image.

[0057] It should be noted that the above-mentioned superposition of at least one second musical score image on top of the original musical score image is not a physical superposition, but rather that the image size of at least one second musical score image corresponds to the image size of the original musical score image, and the position of the notes in at least one second musical score image corresponds to the position of the notes in the original musical score image.

[0058] Optionally, in this embodiment of the application, the original sheet music image is a sheet music image selected by the user.

[0059] Optionally, in the embodiments of this application, the above-mentioned proportional scaling can be to enlarge or reduce the original musical score image.

[0060] Optionally, in the embodiments of this application, the image size of the original musical score image can be larger or smaller than that of at least one second musical score image.

[0061] Optionally, in this embodiment of the application, the original musical score image may be located at the bottom or top layer of a multi-layer musical score image.

[0062] For example, such as Figure 3 As shown, taking the original sheet music image as the bottom layer of a multi-layer sheet music image as an example, after obtaining the original sheet music image 10, the electronic device can perform proportional reduction processing on the original sheet music image 10 to obtain a second sheet music image 11 with a first resolution, and then superimpose the second sheet music image 11 on the original sheet music image 10. Then, the electronic device can perform proportional reduction processing on the second sheet music image 11 again to obtain a third sheet music image 12 with a second resolution, and then superimpose the third sheet music image 12 on the second sheet music image 11. Then, the electronic device can perform proportional reduction processing on the third sheet music image 12 again to obtain a fourth sheet music image 13 with a third resolution, and then superimpose the fourth sheet music image 13 on the third sheet music image 11. In this way, a multi-layer sheet music image is obtained, wherein the image resolution of the original sheet music image is greater than the first resolution, the first resolution is greater than the second resolution, and the second resolution is greater than the third resolution.

[0063] In this embodiment, after obtaining a multi-layered musical score image, the electronic device can input each layer of the musical score image into the recognition module to determine the position of the notes in the musical score image at different resolutions. This improves the flexibility of the electronic device in recognizing musical scores.

[0064] Step 202: The electronic device crops the first musical score image to obtain M first image blocks.

[0065] In this embodiment of the application, each of the M first image blocks contains at least one musical note; M is a positive integer.

[0066] In this embodiment of the application, the electronic device can crop the first musical score image according to a first preset size (e.g., 12×12) and a first order to obtain N image blocks. Then, the electronic device can compare the N image blocks with L labeled image blocks to obtain M first image blocks, where L is a positive integer, N is an integer less than or equal to L, and M is an integer less than or equal to N.

[0067] Optionally, in the embodiments of this application, the first order can be from top to bottom or from left to right. It can be understood that the electronic device crops the first musical score image in the first order to obtain N non-repeating image blocks.

[0068] It should be noted that the above-mentioned labeled image blocks are manually labeled image blocks containing musical notes.

[0069] Specifically, the electronic device can perform an intersection-over-union (IOU) comparison on the position information of N image blocks and L labeled image blocks to obtain the IOU ratio of the N image blocks and L labeled image blocks. Then, the electronic device can obtain M first image blocks through the IOU ratio of the N image blocks and L labeled image blocks.

[0070] For example, after obtaining the intersection-union ratio of N image blocks and L labeled image blocks, the electronic device can identify the image blocks whose intersection-union ratio is greater than a preset threshold (e.g., 0.7) among the N image blocks and L labeled image blocks as the first image blocks.

[0071] Optionally, in the embodiments of this application, step 202 above can be specifically implemented by step 202a below.

[0072] Step 202a: The electronic device crops each layer of the musical score image in the multi-layer musical score image to obtain M first image blocks.

[0073] In this embodiment of the application, the electronic device can crop each layer of the musical score image in the multi-layer musical score image by a first preset size to obtain N image blocks. Then, the electronic device can compare the N image blocks with L labeled image blocks to obtain M first image blocks, where L is a positive integer and N is an integer less than or equal to L.

[0074] It should be noted that the specific implementation process can be found in step 202 above, and will not be repeated here to avoid repetition.

[0075] In this embodiment, the electronic device can crop musical score images of different resolutions to obtain image blocks of different resolutions. The electronic device can then identify these image blocks to determine the positions of notes in the musical score images of different resolutions. This improves the accuracy of the electronic device in identifying notes in musical score images of different resolutions.

[0076] Step 203: The electronic device acquires the first target candidate box for each note in each first image block.

[0077] In this embodiment of the application, the first candidate box of each of the at least one note corresponds to the position information of each note in each first musical score image block.

[0078] In this embodiment of the application, the electronic device can perform box selection processing on each note in each first image block to obtain a first candidate box corresponding to each note, thereby obtaining the first position information of each note based on the first candidate box (that is, the first candidate box corresponds to the position information of each note in each first musical score image block).

[0079] Optionally, in this embodiment of the application, the first position information may be the center coordinate information of the first candidate box; or, the first position information may be the vertex coordinate information of the first candidate box.

[0080] Optionally, in this embodiment, the shape of the first candidate box can be any of the following: square, rectangle, triangle, or parallelogram, etc. Specifically, it can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0081] Optionally, in this embodiment, the size of the first candidate box can be 1×1 or 2×2, etc. Specifically, it can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0082] Optionally, in the embodiments of this application, step 203 can be implemented by step 203a as described below.

[0083] Step 203a: The electronic device inputs M first image blocks into the first recognition module to identify the note positions, so as to output the first position information of each note in each first image block.

[0084] In this embodiment of the application, for each of the M first image blocks, the electronic device can input each first image block into the first recognition module. Then, the first recognition module can perform convolution processing on each first image block to obtain the convolution-processed first image block, and predict and output the first position information of each note in each first image block through a target classifier and a target function.

[0085] In this embodiment, the electronic device can obtain the first position information of the notes in the M first image blocks that are close to those marked by humans by performing three convolution processes on the M first image blocks. In this way, the efficiency of the electronic device in music score recognition can be improved.

[0086] Optionally, in the embodiments of this application, step 203a can be implemented by the following steps 203a1 and 203a2.

[0087] Step 203a1: After the electronic device inputs the i-th first image block into the first recognition module, it determines at least one first candidate box in the i-th first image block based on the first recognition module, and obtains the position information of each first candidate box in the at least one first candidate box on the i-th first image block.

[0088] In this embodiment, the electronic device can predict and output at least one first candidate box containing the position information of each note in each first image block based on the target classifier and the target function.

[0089] Optionally, in this embodiment of the application, the electronic device can predict the classification information of each first candidate box in at least one first candidate box based on the target classifier and the target function.

[0090] Optionally, in this embodiment of the application, the electronic device can indicate the classification of each first candidate box through a target identifier.

[0091] Optionally, in the embodiments of this application, the target identifier can be any of the following: a number identifier, a letter identifier, a special symbol identifier, a combination of number and letter identifier, etc.

[0092] For example, the electronic device can indicate the classification of the first candidate box using 0 and 1, where 0 represents that the first candidate box in the predicted output does not contain a note, and 1 represents that the first candidate box in the predicted output contains a note.

[0093] In this embodiment of the application, although the above embodiment obtains M image blocks containing notes in the first musical score image through the IOU algorithm, there may be image noise in the M image blocks. Therefore, the electronic device can predict and output at least one first candidate box containing the position information of each note in each first image block and the classification information of the first candidate box. In this way, the electronic device can more accurately identify the position of the notes in the first musical score image through the classification information of the first candidate box.

[0094] Optionally, in the embodiments of this application, the above step 203a1 can be specifically implemented by the following steps 203a11 and 203a112.

[0095] Step 203a11: The electronic device performs convolution processing on the i-th first musical score image block to obtain the image features of the i-th first musical score image block.

[0096] In this embodiment of the application, for each of the M first image blocks, the electronic device can input each first image block into the first recognition module. Then, the first recognition module can perform convolution processing on each first image block to obtain the convolution-processed first image block, and predict and output the first position information of each note in each first image block through a target classifier and a target function.

[0097] Specifically, the first recognition module may include three convolutional layers, with each convolutional layer having a 3×3 convolutional kernel, and the number of convolutional layers between each convolutional layer being 10, 16, or 12.

[0098] Step 203a112: The electronic device determines the first candidate box based on the image features of the i-th first musical score image block, and obtains the position information of the first candidate box on the i-th first musical score image block.

[0099] In this embodiment of the application, after obtaining the image features of the i-th first musical score image block, the electronic device can predict and output the first position information of each note in each first image block through a target classifier and a target function.

[0100] Specifically, the target classifier can be a softmax classifier; the objective function can be a bounding box regression function.

[0101] For example, such as Figure 4 As shown, the electronic device can input each first image block 14 into the first recognition module, and then pass it through the first convolutional layer to obtain a 5×5×10 feature map 15 (i.e., feature map, hereinafter referred to as the first feature map). Then, the first feature map is input into the second convolutional layer to obtain a 3×3×16 feature map 16 (hereinafter referred to as the second feature map). Then, the second feature map is input into the third convolutional layer to obtain a 1×1×12 feature map 17 (hereinafter referred to as the third feature map). Then, the electronic device can input the third feature map into the softmax classifier and the bounding box regression function to predict and output the first position information of each note in each first image block. The first two digits of the first feature map, the second feature map and the third feature map represent the image size of the feature map, and the last digit of the first feature map, the second feature map and the third feature map represent the number of features contained in the feature map.

[0102] In this embodiment, the electronic device can obtain the first position information of the notes in the M first image blocks that are close to those marked by humans by performing three convolution processes on the M first image blocks. In this way, the efficiency of the electronic device in music score recognition can be improved.

[0103] Step 203a2: The electronic device, based on the first recognition module, merges at least one first candidate box to obtain a first target candidate box, so as to obtain the first position information of the first target candidate box on the i-th first image block.

[0104] In this embodiment of the application, the i-th first image block is any one of the M first images.

[0105] In this embodiment of the application, since step 203a1 above generates at least one first candidate box of different scales, the electronic device can merge at least one first candidate box using the non-maximum suppression (NMS) algorithm to obtain the first position information of the first target candidate box on the i-th first image block.

[0106] For example, such as Figure 5 As shown in (A), taking a first image patch as an example, the first image patch after passing through the first recognition module contains four first candidate boxes. Since each of the four first candidate boxes indicates an image feature ( Figure 5 (represented by a human face in the image), so as shown in (B) of 5, the electronic device can use the NMS algorithm to merge the four first candidate boxes into one first target candidate box and obtain the first position information of the first target candidate box in the first image.

[0107] In this embodiment, the first recognition module recognizes speed blocks, but the accuracy is not high. Therefore, at least one first candidate box of different scales is generated. The electronic device can filter out at least one first candidate box indicating the same note by using the NMS algorithm. In this way, the electronic device can obtain the position of the note in the i-th first image block more accurately.

[0108] Step 204: The electronic device uses the position of the first candidate box of each note on the first musical score image as the center to crop out the second image block corresponding to each note from the first musical score image.

[0109] In this embodiment of the application, the resolution of the second image block is greater than the resolution of the first image block.

[0110] In this embodiment of the application, after obtaining the first position information of the first target candidate box in each first image block, the electronic device can crop the first musical score image at a second size (e.g., 24×24) according to the center point coordinates of the first position information of each note, so as to obtain the second image block corresponding to each note.

[0111] Specifically, the electronic device can determine the center point coordinates of each note based on the vertex coordinates of the first target candidate box.

[0112] Step 205: The electronic device acquires the second target candidate box for each note in the second image block corresponding to each note.

[0113] In this embodiment of the application, the second candidate box of each of the at least one note corresponds to the position information of each note in the second image block.

[0114] In this embodiment of the application, the electronic device can perform box selection processing on each note in each first image block to obtain a second candidate box corresponding to each note, thereby obtaining the second position information of each note based on the second candidate box.

[0115] Optionally, in this embodiment of the application, the first position information can be the center coordinate information of the note; or, the second position information can be the vertex coordinate information of the note.

[0116] Optionally, in this embodiment, the shape of the second candidate box can be any of the following: square, rectangle, triangle, or parallelogram, etc. Specifically, it can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0117] Optionally, in this embodiment, the size of the second candidate box can be 1×1 or 2×2, etc. Specifically, it can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0118] Optionally, in the embodiments of this application, step 205 can be implemented by step 205a as described below.

[0119] Step 205a: The electronic device inputs the second image block corresponding to each note into the second recognition module to recognize the note position, so as to output the second position information of each note in the second image block.

[0120] In this embodiment of the application, the recognition accuracy of the second recognition module is greater than that of the first recognition module.

[0121] Optionally, in the embodiments of this application, the above step 205a can be implemented by the following steps 205a1 and 205a2.

[0122] Step 205a1: After the electronic device inputs the j-th second image block into the second recognition module, it determines at least one second candidate box in the j-th second image block based on the second recognition module, and obtains the position information of each second candidate box in the at least one second candidate box on the j-th second image block.

[0123] In this embodiment, the electronic device can predict and output at least one second candidate box that contains more accurate positional information of each note in each first image block, based on the target classifier, the target function, and the fully connected layer.

[0124] In this embodiment, since the second recognition module has more convolutional layers and is optimized based on the first candidate box provided by the first recognition module, the electronic device can output more accurate border information than the first recognition module, thus improving the accuracy of the electronic device in recognizing musical scores.

[0125] Optionally, in the embodiments of this application, the above step 205a1 can be specifically implemented by the following steps 205a11 to 205a13.

[0126] Step 205a11: The electronic device performs convolution processing on the j-th second image block to obtain the image features of the j-th second image block.

[0127] In this embodiment of the application, the electronic device can perform convolution processing on the second image block corresponding to each note again to obtain the image features of the j-th second image block.

[0128] Specifically, the second recognition module may include three convolutional layers and fully connected layers, with each convolutional layer having a 3×3 convolutional kernel and the number of convolutional layers between each convolutional layer being 24, 48, or 64.

[0129] It should be noted that, compared to the first recognition module, the resolution of the second image block input by the electronic device to the second recognition module is expanded to 24×24. The second recognition module only processes the second image block containing musical notes. Moreover, compared to the first recognition module, the second recognition module has more convolutional layers and adds fully connected layers. The second recognition module is optimized based on the first candidate box provided by the first recognition module. Therefore, the second recognition module can output more accurate border information than the first recognition module.

[0130] Step 205a12: The electronic device performs a full connection on the image features of the j-th second image block to obtain the fully connected image features.

[0131] In this embodiment of the application, after obtaining the image features of the j-th second image block, the electronic device can input the image features of the j-th second image block into the fully connected layer to obtain the fully connected image features.

[0132] Step 205a13: The electronic device determines the second candidate box based on the fully connected image features and obtains the position information of the second candidate box on the j-th second image block.

[0133] In this embodiment, the electronic device can predict the second position information of each note in the output second image block through a target classifier, a target function, and a fully connected layer.

[0134] In this embodiment, the second recognition module has more convolutional layers and an added fully connected layer. Furthermore, the second recognition module is optimized based on the first candidate box provided by the first recognition module. Therefore, the electronic device can output more accurate border information than the first recognition module, thus improving the accuracy of the electronic device in recognizing musical scores.

[0135] Step 205a2: The electronic device, based on the second recognition module, merges at least one second candidate box to obtain a second target candidate box, so as to obtain the second position information of the second target candidate box on the j-th second image block.

[0136] In this embodiment of the application, the j-th second image block is any one of the second image blocks corresponding to each note.

[0137] In this embodiment of the application, the electronic device can merge at least one second candidate box using the NMS algorithm to obtain the second position information of the second target candidate box on the j-th second image block.

[0138] In this embodiment of the application, the electronic device can use the NMS algorithm to filter out at least one second candidate box indicating the same note. In this way, the electronic device can more accurately determine the position of the note in the j-th second image block.

[0139] Step 206: The electronic device identifies the first musical score image based on the second candidate box for each note.

[0140] Optionally, in this embodiment of the application, the first musical score image includes at least one set of musical staves; the above step 206 can be specifically implemented by the following steps 206a to 206c.

[0141] Step 206a: Based on the position information of the second target candidate box of each note in the second image block, add a first identifier at the position of each note on the first musical score image.

[0142] In this embodiment of the application, the electronic device can divide each set of musical staves in at least one set into K intervals on an average basis, and add a first identifier at the position of each note on the first musical score image in a first order to obtain the target identifier string corresponding to each set of musical staves.

[0143] Optionally, in the embodiments of this application, the first order can be from left to right or from top to bottom.

[0144] Optionally, in the embodiments of this application, the first identifier can be any of the following: a number identifier, a letter identifier, or a special symbol identifier.

[0145] Step 206b: The electronic device acquires and stores the target identifier string corresponding to each set of five-line spectra in at least one set of wireless spectra.

[0146] In this embodiment of the application, the target identifier string corresponding to any one of the at least one set of musical staves is: composed of the first identifier corresponding to each note in any one set of musical staves; the target identifier string corresponding to any one set of musical staves is arranged according to the arrangement order of the notes in any one set of musical staves.

[0147] In this embodiment of the application, after obtaining the target identifier string (hereinafter referred to as the first identifier string) corresponding to each set of musical staff, the electronic device can store the first identifier string in the first application and establish a correspondence between the first identifier string and the musical score information of the first musical score.

[0148] Optionally, in this embodiment of the application, the electronic device can establish a correspondence between the first identifier string and the musical score information of the first musical score through internal or external connections.

[0149] Optionally, in this embodiment of the application, the first application described above may be a database or a server.

[0150] Step 206c: The electronic device identifies the first musical score image based on the target identifier string.

[0151] In this embodiment of the application, after obtaining the target identifier string (hereinafter referred to as the first identifier string) corresponding to each set of musical staff, the electronic device can store the first identifier string in the first application and establish a correspondence between the first identifier string and the musical score information of the first musical score, so that the electronic device can identify the first musical score image according to the target identifier string.

[0152] Optionally, in this embodiment of the application, the electronic device can establish a correspondence between the first identifier string and the musical score information of the first musical score through internal or external connections.

[0153] Optionally, in this embodiment of the application, the first application can be a database or a server.

[0154] In this embodiment, the electronic device establishes a correspondence between the first identifier string and the sheet music information of the first musical score, thereby enabling the electronic device to quickly find the sheet music information corresponding to the first identifier string. This improves the efficiency and convenience of the electronic device in finding sheet music information.

[0155] This application provides a music score recognition method. An electronic device can crop a first music score image to obtain M image blocks. Based on the M first image blocks, a first target candidate box for each note in each of the M first image blocks is obtained. Then, taking the position corresponding to the first target candidate box of each note on the first music score image as the center, a second image block corresponding to each note is cropped from the first music score image. A second target candidate box for each note in the second image block corresponding to each note is obtained. Based on the second candidate box of each note, the first music score image is recognized. The resolution of the second image block is greater than that of the first image block. In this scheme, the electronic device first crops the first musical score image to obtain a first target candidate box for each note, thereby initially identifying the first position information of the notes in the first musical score image. Then, based on the first target candidate box, the first musical score image is cropped a second time to obtain a second image block with higher resolution. Based on the second image block with higher resolution, a second target candidate box with higher accuracy is identified for the notes in the first musical score image. The position information of each note in the second image block in the first musical score image is determined by the second target candidate box. Then, the first musical score image is identified using the second candidate box for each note. Through the second cropping, the electronic device can further accurately identify the position of the notes in the first musical score image. Thus, the musical score can be accurately identified based on the position of the notes in the first musical score image, thereby improving the accuracy of the electronic device in recognizing musical scores.

[0156] It should be noted that the music score recognition method provided in this application can be executed by a music score recognition device, an electronic device, or a functional module or entity within an electronic device. This application uses a music score recognition device executing the music score recognition method as an example to illustrate the music score recognition device provided in this application.

[0157] Figure 6 A schematic diagram of a possible structure of the music score recognition device involved in an embodiment of this application is shown. For example... Figure 6 As shown, the music score recognition device 70 may include: an acquisition module 71, a cropping module 72, and a processing module 73.

[0158] The system includes an acquisition module 71 for acquiring a first musical score image. A cropping module 72 for cropping the first musical score image to obtain M first image blocks, each of which contains at least one musical note; M is a positive integer. The acquisition module 71 is also used to acquire a first target candidate box for each musical note in each first image block, with each first candidate box corresponding to the position information of each musical note in each first image block. The cropping module 72 is also used to crop a second image block corresponding to each musical note from the first musical score image, centered on the position corresponding to the first candidate box of each musical note. The acquisition module 71 is also used to acquire a second target candidate box for each musical note in the second image block corresponding to each musical note, with each second candidate box corresponding to the position information of each musical note in the second image block. The processing module 73 is used to recognize the first musical score image based on the second candidate box of each musical note; wherein the resolution of the second image block is greater than the resolution of the first image block.

[0159] In one possible implementation, the acquisition module 71 is specifically used to input the i-th first musical score image block into the first recognition module, determine at least one first candidate box in the i-th first musical score image block based on the first recognition module, and obtain the position information of each first candidate box in the at least one first candidate box on the i-th first musical score image block; and based on the first recognition module, merge at least one first candidate box to obtain a first target candidate box, and obtain the first position information of the first target candidate box on the i-th first musical score image block; wherein, the i-th first musical score image block is any one of the M first musical score images; the acquisition module... The block is specifically used to input the j-th second image block into the second recognition module, determine at least one second candidate box in the j-th second image block based on the second recognition module, and obtain the position information of each second candidate box in the at least one second candidate box on the j-th second image block; and based on the second recognition module, merge at least one second candidate box to obtain a second target candidate box, and obtain the second position information of the second target candidate box on the j-th second image block; wherein, the j-th second image block is any one of the second image blocks corresponding to each note; wherein, the recognition accuracy of the second recognition module is greater than the recognition accuracy of the first recognition module.

[0160] In one possible implementation, the acquisition module 71 is specifically used to perform convolution processing on the i-th first musical score image block to obtain the image features of the i-th first musical score image block; and based on the image features of the i-th first musical score image block, determine the first candidate box and obtain the position information of the first candidate box on the i-th first musical score image block.

[0161] In one possible implementation, the acquisition module 71 is specifically used to perform convolution processing on the j-th second image block to obtain the image features of the j-th second image block; and to perform a full connection on the image features of the j-th second image block to obtain the fully connected image features; and to determine the second candidate box based on the fully connected image features, and to obtain the position information of the second candidate box on the j-th second image block.

[0162] In one possible implementation, the acquisition module 71 is specifically used to construct a multi-layered musical score image based on the original musical score image; wherein, the multi-layered musical score image is a first musical score image; and each layer of the multi-layered musical score image has a different image size. The cropping module 72 is specifically used to crop each layer of the multi-layered musical score image to obtain M first image blocks.

[0163] In one possible implementation, the first musical score image includes at least one set of musical staves; the processing module 73 is specifically used to acquire the position information of the second target candidate box of each note based on the acquisition module, add a first identifier at the position of each note on the first musical score image; acquire and store the target identifier string corresponding to each set of musical staves in at least one set of musical staves; and identify the first musical score image using the target identifier string; wherein, the target identifier string corresponding to any musical stave in at least one set of musical staves is composed of the first identifier corresponding to each note in any musical stave; and the target identifier string corresponding to any musical stave is arranged according to the arrangement order of the notes in any musical stave.

[0164] This application provides a music score recognition device. The device performs a first cropping of a first music score image to obtain a first target candidate box for each note, thereby initially identifying the first position information of the notes in the first music score image using the first target candidate boxes. Then, based on the first target candidate boxes, the first music score image is cropped a second time to obtain a second image block with higher resolution. Based on this higher-resolution second image block, second target candidate boxes with higher accuracy are identified for the notes in the first music score image. The position information of each note in the second image block within the first music score image is determined using the second target candidate boxes. Furthermore, the first music score image is recognized using the second candidate boxes for each note. Thus, by performing a second cropping, the electronic device can further accurately identify the position of notes in the first music score image, thereby accurately recognizing the music score based on the position of the notes in the first music score image. This improves the accuracy of the music score recognition device in recognizing music scores.

[0165] The music score recognition device in this application embodiment can be a device, or a component, integrated circuit, or chip in an electronic device. The device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0166] The music score recognition device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0167] The music score recognition device provided in this application embodiment can achieve... Figures 1 to 5 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0168] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores a program or instructions that can run on the processor 91. When the program or instructions are executed by the processor 91, they implement the various steps of the above-described music score recognition method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0169] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0170] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0171] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0172] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0173] The processor 110 acquires a first musical score image; crops the first musical score image to obtain M first image blocks, each of the M first image blocks containing at least one musical note; M is a positive integer; acquires a first target candidate box for each musical note in each first image block, the first candidate box for each musical note corresponding to the position information of each musical note in each first musical score image block; crops a second image block corresponding to each musical note in the first musical score image, centered on the position corresponding to the first candidate box for each musical note in the first musical score image; acquires a second target candidate box for each musical note in the second image block corresponding to each musical note, the second candidate box for each musical note corresponding to the position information of each musical note in the second image block; and performs recognition on the first musical score image based on the second candidate box for each musical note; wherein the resolution of the second image block is greater than the resolution of the first musical score image block.

[0174] This application provides an electronic device that performs a first cropping of a first musical score image to obtain a first target candidate box for each note, thereby initially identifying the first position information of the notes in the first musical score image using the first target candidate boxes. Then, based on the first target candidate boxes, the first musical score image is cropped a second time to obtain a second image block with higher resolution. Based on the second image block with higher resolution, a second target candidate box with higher accuracy is identified for the notes in the first musical score image. The position information of each note in the second image block in the first musical score image is determined using the second target candidate boxes. Then, the first musical score image is identified using the second candidate boxes for each note. In this way, through secondary cropping, the electronic device can further accurately identify the position of the notes in the first musical score image, thereby accurately identifying the musical score based on the position of the notes in the first musical score image, thus improving the accuracy of the electronic device in recognizing musical scores.

[0175] Optionally, in this embodiment of the application, the processor 110 inputs the i-th first image block into the first recognition module, determines at least one first candidate box in the i-th first image block based on the first recognition module, and obtains the position information of each first candidate box in the at least one first candidate box on the i-th first image block; based on the first recognition module, merges the at least one first candidate box to obtain a first target candidate box, and obtains the first position information of the first target candidate box on the i-th first image block; wherein, the i-th first image block is any one of M first images; and inputs the j-th second image block into the second recognition module, determines at least one second candidate box in the j-th second image block based on the second recognition module, and obtains the position information of each second candidate box in the at least one second candidate box on the j-th second image block; based on the second recognition module, merges the at least one second candidate box to obtain a second target candidate box, and obtains the second position information of the second target candidate box on the j-th second image block; wherein, the j-th second image block is any one of the second image blocks corresponding to each note; wherein, the recognition accuracy of the second recognition module is greater than the recognition accuracy of the first recognition module.

[0176] Optionally, in this embodiment of the application, the processor 110 is specifically used to perform convolution processing on the i-th first musical score image block to obtain the image features of the i-th first musical score image block; determine the first candidate box based on the image features of the i-th first musical score image block, and obtain the position information of the first candidate box on the i-th first musical score image block.

[0177] Optionally, in this embodiment of the application, the processor 110 is specifically used to perform convolution processing on the j-th second image block to obtain the image features of the j-th second image block; to perform full connection on the image features of the j-th second image block to obtain the fully connected image features; to determine the second candidate box based on the fully connected image features, and to obtain the position information of the second candidate box on the j-th second image block.

[0178] Optionally, in this embodiment of the application, the processor 110 is specifically used to construct a multi-layered musical score image based on the original musical score image; wherein, the multi-layered musical score image is a first musical score image; each layer of the multi-layered musical score image has a different image size; and each layer of the multi-layered musical score image is cropped to obtain M first image blocks.

[0179] Optionally, in this embodiment of the application, the first musical score image includes at least one set of musical staves; the processor 110 is specifically configured to add a first identifier to the position of each note in the first musical score image based on the position information of the second target candidate box of each note in the second image block; acquire and store the target identifier string corresponding to each set of musical staves in at least one set of musical staves; and identify the first musical score image according to the target identifier string; wherein, the target identifier string corresponding to any musical stave in at least one set of musical staves is composed of the first identifier corresponding to each note in any musical stave; and the target identifier string corresponding to any musical stave is arranged according to the arrangement order of the notes in any musical stave.

[0180] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0181] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0182] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0183] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0184] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0185] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0186] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0187] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0188] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0189] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described music score recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0190] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0192] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for music score recognition, characterized in that, The method includes: Obtain the first musical score image; The first musical score image is cropped to obtain M first image blocks, each of the M first image blocks containing at least one musical note; M is a positive integer. Obtain a first target candidate box for each note in each first image block, wherein the first candidate box for each note corresponds to the position information of each note in each first image block; Using the position of the first candidate box of each note on the first musical score image as the center, the second image block corresponding to each note is cropped from the first musical score image; Obtain the second target candidate box for each note in the second image block corresponding to each note, wherein the second candidate box for each note corresponds to the position information of each note in the second image block; The first musical score image is identified based on the second candidate box for each note; The resolution of the second image block is greater than that of the first image block.

2. The method according to claim 1, characterized in that, The step of obtaining the first target candidate box for each note in each first image block includes: After the i-th first image block is input into the first recognition module, at least one first candidate box is determined in the i-th first image block based on the first recognition module, and the position information of each first candidate box in the at least one first candidate box on the i-th first image block is obtained; Based on the first recognition module, the at least one first candidate box is merged to obtain a first target candidate box, and the first position information of the first target candidate box on the i-th first image block is obtained; Wherein, the i-th first image block is any one of the M first image blocks; The step of obtaining the second target candidate box for each note in the second image block corresponding to each note includes: After the j-th second image block is input into the second recognition module, at least one second candidate box is determined in the j-th second image block based on the second recognition module, and the position information of each second candidate box in the at least one second candidate box on the j-th second image block is obtained; Based on the second recognition module, the at least one second candidate box is merged to obtain a second target candidate box, and the second position information of the second target candidate box on the j-th second image block is obtained; Wherein, the j-th second image block is any one of the second image blocks corresponding to each note; The recognition accuracy of the second recognition module is greater than that of the first recognition module.

3. The method according to claim 2, characterized in that, The step of determining at least one first candidate box in the i-th first image block based on the first recognition module, and obtaining the position information of each first candidate box in the at least one first candidate box on the i-th first image block, includes: The image features of the i-th first image block are obtained by performing convolution processing on the i-th first image block; Based on the image features of the i-th first image block, the first candidate box is determined, and the position information of the first candidate box on the i-th first image block is obtained.

4. The method according to claim 2, characterized in that, After inputting the j-th second image block into the second recognition module, the second recognition module determines at least one second candidate box in the j-th second image block, and obtains the position information of each second candidate box in the at least one second candidate box on the j-th second image block, including: The j-th second image block is convolved to obtain the image features of the j-th second image block; Perform a full connection on the image features of the j-th second image block to obtain the fully connected image features; Based on the fully connected image features, the second candidate box is determined, and the position information of the second candidate box on the j-th second image block is obtained.

5. The method according to claim 1, characterized in that, The acquisition of the first musical score image includes: Construct multi-layered musical score images based on the original musical score images; Wherein, the multi-layered musical score image is the first musical score image; Each layer of the multi-layered musical score image has a different image size; The step of cropping the first musical score image to obtain M first image blocks includes: Each layer of the musical score image in the multi-layered musical score image is cropped to obtain M first image blocks.

6. The method according to claim 1, characterized in that, The first musical score image includes at least one set of musical staves; the step of identifying the first musical score image based on the second candidate box of each note includes: Based on the position information of the second target candidate box for each note in the second image block, a first identifier is added at the position of each note on the first musical score image. Acquire and store the target identifier string corresponding to each set of five-line staff in the at least one set of wireless spectra; The first musical score image is identified based on the target identifier string; Wherein, the target identifier string corresponding to any one of the at least one set of staffs is composed of the first identifier corresponding to each note in the staff; the target identifier string corresponding to any one staff is arranged according to the arrangement order of the notes in the staff.

7. A music score recognition device, characterized in that, The music score recognition device includes: an acquisition module, a cropping module, and a processing module; The acquisition module is used to acquire the first musical score image; The cropping module is used to crop the first musical score image to obtain M first image blocks, each of the M first image blocks containing at least one musical note; M is a positive integer; The acquisition module is further configured to acquire a first target candidate box for each note in each first image block, wherein the first candidate box for each note corresponds to the position information of each note in each first image block; The cropping module is further configured to crop out the second image block corresponding to each note from the first musical score image, centered on the position of the first candidate box of each note on the first musical score image; The acquisition module is further configured to acquire a second target candidate box for each note in the second image block corresponding to each note, wherein the second candidate box for each note corresponds to the position information of each note in the second image block; The processing module is used to identify the first musical score image based on the second candidate box of each note; The resolution of the second image block is greater than that of the first image block.

8. The apparatus according to claim 7, characterized in that, The acquisition module is specifically used to input the i-th first image block into the first recognition module, and then, based on the first recognition module, determine at least one first candidate box in the i-th first image block, and obtain the position information of each first candidate box in the at least one first candidate box on the i-th first image block; Based on the first recognition module, the at least one first candidate box is merged to obtain a first target candidate box, and the first position information of the first target candidate box on the i-th first image block is obtained; wherein, the i-th first image block is any one of the M first image blocks; The acquisition module is specifically configured to input the j-th second image block into the second recognition module, determine at least one second candidate box in the j-th second image block based on the second recognition module, and obtain the position information of each second candidate box in the at least one second candidate box on the j-th second image block; and merge the at least one second candidate box based on the second recognition module to obtain a second target candidate box, and obtain the second position information of the second target candidate box on the j-th second image block; wherein, the j-th second image block is any one of the second image blocks corresponding to each note; The recognition accuracy of the second recognition module is greater than that of the first recognition module.

9. The apparatus according to claim 8, characterized in that, The acquisition module is specifically used to perform convolution processing on the i-th first image block to obtain the image features of the i-th first image block; and to determine the first candidate box based on the image features of the i-th first image block, and to obtain the position information of the first candidate box on the i-th first image block.

10. The apparatus according to claim 8, characterized in that, The acquisition module is specifically used to perform convolution processing on the j-th second image block to obtain the image features of the j-th second image block; and to perform a fully connected operation on the image features of the j-th second image block to obtain the fully connected image features; and to determine the second candidate box based on the fully connected image features, and to obtain the position information of the second candidate box on the j-th second image block.

11. The apparatus according to claim 7, characterized in that, The acquisition module is specifically used to construct a multi-layered musical score image based on the original musical score image; wherein, the multi-layered musical score image is the first musical score image; and each layer of the multi-layered musical score image has a different image size; The cropping module is specifically used to crop each layer of the musical score image in the multi-layer musical score image to obtain M first image blocks.

12. The apparatus according to claim 7, characterized in that, The first musical score image includes at least one set of musical staves; The processing module is specifically used to obtain the position information of the second target candidate box of each note based on the acquisition module, and add a first identifier at the position of each note on the first musical score image. Acquire and store the target identifier string corresponding to each set of five-line staff in the at least one set of wireless spectra; The target identifier string is used to identify the first musical score image; wherein, the target identifier string corresponding to any one of the at least one set of musical staves is composed of the first identifier corresponding to each note in the musical staves; the target identifier string corresponding to any one musical stave is arranged according to the arrangement order of the notes in the musical staves.

13. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the music score recognition method as described in any one of claims 1 to 6.

14. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the music score recognition method as described in any one of claims 1 to 6.

Images