AI music score processing method, computer device, and computer storage medium

Abstract

Embodiments of the present application disclose an AI music score processing method, a computer device and a computer storage medium. The computer device inputs a target music score into a target music score element detection model to obtain multiple elements of the target music score and label information of the multiple elements, obtains overall structured information of the target music score according to the multiple elements and the label information of the multiple elements, the overall structured information of the target music score being a music measure structure of the target music score, further obtains lyric information of the target music score, the lyric information including lyric text and lyric time information, and adjusts the overall structured information of the target music score according to the lyric time information, so that the lyric time information matches the music measure structure of the target music score. Therefore, by extracting and reproducing each element of the target music score, the structure and content of each music measure of an accompaniment score are obtained, so that the target music score is converted into the accompaniment score which is convenient for a user to play a musical instrument, and the use experience of the user for a music application program is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of music score picture processing, and in particular to an AI music score processing method, a computer device and a computer storage medium. BACKGROUND

[0002] A user plays a song using a music software client, and the music software client provides a music score viewing function, and the user can use this function to view the music score of the song being played.

[0003] However, the music score of the song is displayed in the form of a picture, and the user views the music score picture, but cannot generate a corresponding accompaniment score. For example, the user needs to view the accompaniment score when using the music software client, and hopes to use the accompaniment score to play a musical instrument, such as playing a guitar score to sing, but the current music software client cannot provide this function, and only supports the user to view the music score, and cannot support the user to obtain the accompaniment score based on the music score. SUMMARY

[0004] Embodiments of the present application provide an AI music score processing method, a computer device and a computer storage medium, for generating an editable music score file based on a music score picture.

[0005] The first aspect of the embodiments of the present application provides an AI music score processing method, and the method comprises:

[0006] inputting a target music score to be processed into a pre-trained target music score element detection model to obtain a plurality of elements of the target music score and label information of the plurality of elements;

[0007] obtaining overall structured information of the target music score according to the plurality of elements and the label information of the plurality of elements, the overall structured information of the target music score being a music measure structure of the target music score;

[0008] obtaining lyrics information of the target music score, the lyrics information comprising lyrics text and lyrics time information;

[0009] adjusting the overall structured information of the target music score according to the lyrics time information, so that the lyrics time information matches the music measure structure of the target music score.

[0010] The second aspect of the embodiments of the present application provides a computer device, and the computer device comprises:

[0011] a first obtaining unit configured to input a target music score to be processed into a pre-trained target music score element detection model to obtain a plurality of elements of the target music score and label information of the plurality of elements;

[0012] a first processing unit configured to obtain overall structured information of the target musical score according to the multiple elements and the label information of the multiple elements, the overall structured information of the target musical score being a musical bar structure of the target musical score;

[0013] a second obtaining unit configured to obtain lyrics information of the target musical score, the lyrics information including lyrics text and lyrics time information;

[0014] an adjusting unit configured to adjust the overall structured information of the target musical score according to the lyrics time information, so that the lyrics time information matches the musical bar structure of the target musical score.

[0015] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer storage medium, the computer storage medium stores instructions, and the instructions cause a computer to execute the method of the first aspect when executed on the computer.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product causes a computer device to execute the method of the first aspect when running on the computer device.

[0018] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0019] In the embodiment, the target musical score is input into a target musical score element detection model that is pre-trained to obtain multiple elements of the target musical score and label information of the multiple elements, and the overall structured information of the target musical score is obtained according to the multiple elements and the label information of the multiple elements, the overall structured information of the target musical score being a musical bar structure of the target musical score. Then, the lyrics information of the target musical score is obtained, the lyrics information including lyrics text and lyrics time information, and the overall structured information of the target musical score is adjusted according to the lyrics time information, so that the lyrics time information matches the musical bar structure of the target musical score. Therefore, by extracting and reproducing each element of the target musical score, the structure and content of each musical bar of the accompaniment score are obtained, so that the target musical score is converted into the accompaniment score which is convenient for users to play musical instruments, and the use experience of the user for the music application program is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 FIG. 1 is a schematic diagram of a network framework in an embodiment of the present application;

[0021] Figure 2 FIG. 2 is a schematic diagram of an AI musical score processing method in an embodiment of the present application;

[0022] Figure 3 Another flowchart of an AI music score processing method in an embodiment of the present application;

[0023] Figure 4 A network structure diagram of a target music score element detection model in an embodiment of the present application;

[0024] Figure 5 A performance mode diagram of an element detection result of a target music score picture in an embodiment of the present application;

[0025] Figure 6 An exemplary diagram of an English music score fragment in an embodiment of the present application;

[0026] Figure 7 A correction mode diagram of a music lyrics detection box in an embodiment of the present application;

[0027] Figure 8 An exemplary diagram of converting a target music score into an accompaniment score in an embodiment of the present application;

[0028] Figure 9 A structure diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application provide an AI music score processing method, a computer device and a computer storage medium, which are used for generating an editable music score file based on a music score picture.

[0030] Please refer to Figure 1 The network framework in the embodiments of the present application includes:

[0031] The business server 100 and a terminal cluster; the terminal cluster can include terminal devices 200a, 200b, 200c, …, 200n, and the like.

[0032] The business server 100 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud database, cloud service, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform. The terminal devices (including terminal device 200a, terminal device 200b, terminal device 200c,..., and terminal device 200n) can be smart phones, tablet computers, notebook computers, desktop computers, palm computers, mobile internet devices (MID), wearable devices (such as smart watches, smart bracelets, etc.), smart computers, and smart vehicles.

[0033] The business server 100 can establish a communication connection with each terminal device in the terminal cluster, and each terminal device in the terminal cluster can also establish a communication connection. In other words, the business server 100 can establish a communication connection with each terminal device in the terminal device 200a, terminal device 200b, terminal device 200c,..., and terminal device 200n. For example, a communication connection can be established between the terminal device 200a and the business server 100. A communication connection can be established between the terminal device 200a and the terminal device 200b, and a communication connection can also be established between the terminal device 200a and the terminal device 200c. The communication connection is not limited to the connection method, which can be directly or indirectly connected through wired communication or wireless communication, and the specific connection method can be determined according to the actual application scenario.

[0034] It should be understood that each terminal device in the terminal cluster as shown in Figure 1 The application client can be installed on each terminal device in the terminal cluster. When the application client runs in each terminal device, it can respectively interact with the business server 100 to enable the business server 100 to receive business data (such as user identity data uploaded by the user through the terminal device) from each terminal device. The application client can be a music playing application, a karaoke software application, a browser application, a social application, an instant messaging application, a live broadcast application, a game application, a short video application, a video application, a shopping application, a novel application, a payment application, etc. with the function of displaying text, image, audio, and video data information. The specific application scenario can be determined, and the application client can be a standalone client or an embedded sub-client integrated in a certain client (such as a music application or a karaoke software application).

[0035] The following will be described in combination withFigure 1 The network framework described herein is used to illustrate the AI ​​music score processing method in the embodiments of this application:

[0036] Please see Figure 2 One embodiment of the AI ​​music score processing method in this application includes:

[0037] 201. Input the target musical score to be processed into the pre-trained target musical score element detection model to obtain multiple elements of the target musical score and the label information of the multiple elements;

[0038] The method of this embodiment can be applied to a computer device, which may be... Figure 1 The network framework shown includes a service server 100 or various terminal devices. Alternatively, the method in this embodiment can be derived from... Figure 1 The business server 100 and each terminal device in the network framework shown execute together.

[0039] When processing the target musical score, the computer device obtains a pre-trained target musical score element detection model. This model is trained by a machine learning algorithm on multiple sets of training samples, and each set of training samples includes a musical score image and label information for each element in the musical score image. The musical score image contains various elements; for example, for tablature, it includes bar lines, tablature areas, chords, lyrics, tuplets, rests, notes, dotted notes, repeat signs, and heading information, among others. The label information is used to distinguish the category of an element, indicating which of the aforementioned categories it belongs to.

[0040] Therefore, the trained target musical score element detection model has learned the features of each element in the musical score image and its corresponding classification information, and can thus be applied in this embodiment to identify each element in the musical score image and its classification.

[0041] 202. Based on the multiple elements and the tag information of the multiple elements, the overall structured information of the target musical score is obtained, wherein the overall structured information of the target musical score is the musical measure structure of the target musical score;

[0042] In this embodiment, the computer device can generate usable overall structured information of the target score based on the semantic information in the target score (i.e., the elements of the target score and the label information of each element) and the music theory rules preset by the user. This overall structured information is used to represent the score structure of the target score, specifically including the musical measure structure of the target score. Each musical measure structure includes any information in the musical measure in music theory, such as bar lines, dotted notes, tuplets, notes, time signatures, BPM, chords, repeat signs, and lyrics.

[0043] 203、obtaining lyric information of the target musical score, the lyric information comprising lyric texts and lyric time information;

[0044] To facilitate matching of the lyrics of the target musical score with the structure of the target musical score, the computer device further obtains lyric information of the target musical score, the lyric information comprising lyric texts and lyric time information. The lyric time information, i.e., lyric timestamps, indicates corresponding playback start time points and playback end time points of each sentence of lyrics and each character in each sentence of lyrics in the song. When playing the song in the music playing application program, the music playing application program can display, in a song playing interface, lyrics corresponding to the playback progress of the current song according to the lyric texts, such as the QRC lyric texts.

[0045] 204、adjusting the overall structured information of the target musical score according to the lyric time information, so that the lyric time information matches the musical measure structure of the target musical score;

[0046] After obtaining the lyric information of the target musical score, the overall structured information of the target musical score can be adjusted according to the lyric time information of each sentence of lyrics or the lyric time information of each character of lyrics. For example, the time stamp of a lyric is determined to determine the corresponding note of the lyric, and the time stamp of the corresponding note of the lyric is determined, so that the corresponding lyrics and score sections are displayed according to the played sections when a user plays an instrument based on the accompaniment score, i.e., the lyric time information of each sentence of lyrics and the lyric time information of each character of lyrics match the musical measure structure of the target musical score, facilitating the user to view the accompaniment score and play the instrument based on the accompaniment score.

[0047] In this embodiment, the target musical score is input into the target musical score element detection model that is pre-trained to obtain multiple elements of the target musical score and label information of the multiple elements. The overall structured information of the target musical score is obtained according to the multiple elements and the label information of the multiple elements, and the overall structured information of the target musical score is the musical measure structure of the target musical score. The lyric information of the target musical score is further obtained, the lyric information comprising lyric texts and lyric time information. The overall structured information of the target musical score is adjusted according to the lyric time information, so that the lyric time information matches the musical measure structure of the target musical score. Therefore, by extracting and reproducing each element of the target musical score, the structure and content of each musical measure of the accompaniment score are obtained, so that the target musical score is converted into the accompaniment score for facilitating the user to play the instrument, and the user experience of using the music application program is improved.

[0048] The foregoing Figure 2 The foregoing Figure 3 The AI musical score processing method in the embodiment of the present application includes:

[0049] 301. Input the target musical score to be processed into the pre-trained target musical score element detection model to obtain multiple elements of the target musical score and the label information of the multiple elements;

[0050] In this embodiment, the target musical score element detection model can be any neural network model. In a preferred embodiment, the target musical score element detection model can be the DBnet model (Real-time Scene Text Detection with Differentiable Binarization). The principle of the DBnet model is based on a segmentation algorithm. First, the network outputs a probability map of text segmentation. Then, a set threshold is used to convert the probability map into a binary map. Finally, the detection result (detection box coordinates) is obtained through post-processing.

[0051] Specifically, the model structure of the target musical score element detection model in this embodiment is as follows: Figure 4 As shown, the network structure of the target musical score element detection model includes a Feature Pyramid Network (FPN), i.e., the backbone structure shown in the figure, as well as an upsampling network connected to the Feature Pyramid Network (i.e., the header structure shown in the figure), a concat layer connected to the upsampling network, and a first convolutional network, a second convolutional network, and a third convolutional network connected to the concat layer, respectively, and a DB layer connected to the first and second convolutional networks. This embodiment of the target musical score element detection model, based on the DBnet model's structure, adds a network structure (i.e., the aforementioned third convolutional network) for identifying the element category to which the detection box of each element in the target musical score belongs, making it more suitable for this embodiment.

[0052] Of course, in addition to setting the network structure of the target musical score element detection model in this embodiment based on the DBnet model, it can also be set based on the upgraded version of the DBnet model, DBnet++, or based on the network structure of MaskR-CNN. This embodiment does not limit this.

[0053] During model training, personnel can label the positions of each element in the sheet music image used for model training and their corresponding categories, such as labeling the positions of lyrics and their labels (i.e., lyric labels), labeling the positions of bar lines and their labels, and so on. Therefore, during model training, the target sheet music element detection model can learn the features of each element in the sheet music image and its corresponding classification information.

[0054] When detecting and recognizing the target musical score, the target musical score element detection model performs the following operations:

[0055] Extract at least one feature map from the target musical score;

[0056] Upsample the at least one feature map to a preset size, and perform a concat operation on the at least one feature map of the preset size to obtain the target feature map;

[0057] Predict the probability that each pixel in the target feature map belongs to the text, and obtain the probability map corresponding to the target feature map;

[0058] Predict the threshold of each pixel in the target feature map to obtain the threshold map corresponding to the target feature map;

[0059] Differentiable binarization is performed on the probability map and the threshold map to obtain the target binary image, which includes the values ​​corresponding to multiple pixels.

[0060] In the target binary image, the location of the region containing multiple pixels with preset values ​​is determined as the position of the element;

[0061] The target feature map is identified to determine the label information of each element in the target musical score.

[0062] For example, in Figure 4 In the model structure shown, the feature map pyramid network extracts at least one feature map of the target musical score; the upsampling network upsamples at least one feature map to a preset size; the concat layer performs a concat operation on at least one feature map of the preset size to obtain the target feature map; the first convolutional network predicts the probability that each pixel in the target feature map belongs to text, obtaining the probability map corresponding to the target feature map; the second convolutional network predicts the threshold of each pixel in the target feature map, obtaining the threshold map corresponding to the target feature map; the DB layer performs differentiable binarization processing (i.e., DB operation, differentiable binarization) on the probability map and the threshold map to obtain the target binary image, and in the target binary image, the positions of the regions where multiple pixels with preset values ​​are located are determined as the positions of elements; the third convolutional network identifies the target feature map to determine the label information of each element in the target musical score, that is, the label information represents the classification result of the elements in the target musical score.

[0063] For example, if each pixel in the target binary image has a value of 0 or 1, the region containing multiple pixels with a value of 1 can be determined. Then, the outline of the region can be drawn with line segments, and the position of the region can be determined as the position of the element.

[0064] In this process, at least one feature map extracted by the feature map pyramid network is upsampled, which can be upsampled to 1 / 4 of the size of the original sheet music image, or upsampled to other sizes.

[0065] In a preferred embodiment, in order to address the problem of excessive regional differences among the elements detected in the target musical score, multiple target musical score element detection models can be used to detect and identify the elements of the target musical score. The model structures of these multiple target musical score element detection models can be the same, but the network weights are different, and the target musical score will be resampled to different sizes when it enters these multiple target musical score element detection models.

[0066] For example, for tablature, three target musical element detection models can be set up. Model 1 is used to detect the tablature region in the musical score image, Model 2 is used to detect the note region and header information in the musical score image, and Model 3 is used to detect other regions besides header information, notes, and tablature, such as lyrics, bar lines, dotted notes, etc. The network structure of these three models is consistent, for example, they can all be the network model structure of the target musical element detection models mentioned above, but the network weights between them are different, and the target musical score will be resampled to different sizes in these three models. For example, in Model 1 and Model 2, the image will be resampled to 4068*1024, and in Model 3 it will be resampled to 4068*3072.

[0067] 302. Based on the multiple elements and the tag information of the multiple elements, the overall structured information of the target musical score is obtained, wherein the overall structured information of the target musical score is the musical measure structure of the target musical score;

[0068] In this embodiment, the overall structured information of the target score is obtained based on multiple elements of the target score and the label information of those multiple elements. One way to do this is to assign lyrics, chords, and sustains to measures in the accompaniment score based on the positional distribution information of bar lines and notes in the multiple elements of the target score. For example, if the vertical coordinate of a certain lyric is between the vertical coordinate of the first measure and the vertical coordinate of the second measure, and since the lyrics are below the measure, then the lyrics are determined to correspond to the first measure. The first measure includes multiple measures. If the horizontal coordinate of the lyrics is within the horizontal coordinate range of the second measure, or if most of the horizontal coordinate range of the lyrics (e.g., more than 60%) is within the horizontal coordinate range of the second measure, then the lyrics are determined to correspond to the second measure in the first measure. Then, missing or incorrect measures in the accompaniment score can be processed based on the song's rhythm information. Misidentified data in the measures can be removed based on the position information of the elements. The correspondence between lyrics and notes within the rhythm can be processed. The repeated measures can be processed by combining the repeat markers and the multiple lines of lyrics within the measures of the target score to obtain the overall structured information of the target score.

[0069] For example, the missing or incorrect measures in the accompaniment score can be processed according to the beat information in the header information of the score, for example, if the bar line is not recognized or is recognized more than once, some errors can be corrected according to the song beat (such as 4 / 4 beat, that is, one measure is 4 four-four notes) and the total length of the notes in the current recognized measure. If the length of the notes in the current recognized measure is 8 four-four notes, it is obvious that the bar line is missed; if the length of the notes in the measure is much lower than the length defined by the song beat, it is possible that there are more misrecognized bar lines that need to be removed.

[0070] Invalid misrecognized data in the measure can also be removed according to the position information of the elements in the measure. For example, the score picture can have annotation information such as male voice singing and female voice singing. Since the notes or lyrics are fixed in a line, the annotation information is generally marked in the upper right corner or upper left corner of the line where the notes are located, and is not in the same line as the notes. Therefore, it can be identified whether the information in the line where the notes are located is in the same line as the line where the notes are located. Elements that are not in the same line can be removed, such as the above-mentioned annotation information of male voice singing and female voice singing.

[0071] In one embodiment, the repeated measures are processed in combination with the repeated identification and the multiple-line lyrics in the measures of the target score. The lyrics time information of the multiple-line lyrics in the measures of the target score is matched with the time information corresponding to the measures of the target score to determine the position information of the lyrics. The corresponding relationship between the measures of the accompaniment score and the lyrics in the measures of the accompaniment score is adjusted based on the repeated identification to obtain the overall structured information of the accompaniment score based on the target score.

[0072] For example, the measures of the repetition can be copied and placed in the corresponding position according to the definition of the repeated identification, and the multiple-line lyrics can also be split accordingly so that each measure corresponds to a line of lyrics. The purpose is to make the overall measure structure of the score continuous and unidirectional (that is, consistent with the singing order). For example, if the target score has two repetitions of "measure A / measure B", the accompaniment score after processing shows "measure A / measure B / measure A / measure B".

[0073] 303, obtain lyrics information of the target score, the lyrics information including lyrics text and lyrics time information;

[0074] In this embodiment, the target score element detection model identifies the position of each element in the target score and the classification result of the element according to the characteristics of each element learned in the model training process, and then outputs the detection result of the target score, that is, the detection result includes the position information of the element in the target score and the label information representing the classification result of the element.

[0075] Therefore, after the positions of the elements are determined, a rectangular frame can be used to frame the area where each element is located. Therefore, the position information of each element in the target musical score includes a detection frame surrounding the position of the element in the target musical score. The computer device can establish a coordinate system in the target musical score, and the coordinates of the reference point of the detection frame of each element can be determined based on the coordinate system in the target musical score, where the reference point is any point in the detection frame. For example, a coordinate system is established with the upper left corner of the target musical score as the origin, the straight line from the upper left corner to the upper right corner as the horizontal coordinate axis, and the straight line from the upper left corner to the lower left corner as the vertical coordinate axis, and the reference point can be the point of the upper left corner of the detection frame, and the coordinates of the reference point can be determined in the coordinate system.

[0076] As shown in a music score picture, after the detection processing of the target musical score element detection model, the detection frames of various elements in the music score picture can be determined, including the detection frame of the lyrics, the detection frame of the musical score header information, the detection frame of the six-line staff, and the detection frames of various elements. Further, based on the coordinate system of the music score picture, the coordinates of the reference point of each element detection frame can be determined, and the coordinates can be the coordinates of the upper left corner of the detection frame, or any point on the edge of the detection frame as the coordinates of the detection frame. Figure 5

[0077] After obtaining the detection frames of the elements in the target musical score, the detection frames may

[0078] In a preferred embodiment of the detection frame correction, the lyrics detection frame can be corrected. Specifically, in the coordinate system, the point with the maximum vertical coordinate and the point with the minimum vertical coordinate in each lyrics detection frame are determined, the difference between the maximum vertical coordinate and the minimum vertical coordinate is taken as the height of the lyrics detection frame, the average height of all lyrics detection frames of the target musical score is counted, the multiple lyrics detection frames are sorted in ascending or descending order of the vertical coordinates of the reference points to obtain a lyrics detection frame sequence, the vertical coordinates of the reference points of each adjacent two lyrics detection frames in the lyrics detection frame sequence are subtracted, and if the difference is less than or equal to a preset percentage of the average height, it is determined that the two lyrics detection frames with the vertical coordinate difference are in the same row; if the difference is greater than the preset percentage of the average height, the lyrics detection frame with the smaller vertical coordinate of the reference point in the two lyrics detection frames with the vertical coordinate difference is taken as the lyrics detection frame of the next row.

[0079] For example, in Figure 5 ​In the shown musical score, after determining the detection frames of lyrics such as "I", "sea", and "eye", the average height of all the lyrics detection frames is calculated. The height of a lyrics detection frame can be the difference between the maximum ordinate and the minimum ordinate of the lyrics detection frame. Then, the multiple lyrics detection frames are sorted in ascending or descending order according to the ordinates of the reference points to obtain a sequence of lyrics detection frames, such as comparing and sorting the ordinates of the points at the upper left corners of each lyrics detection frame. Next, the difference is calculated between the ordinates of the reference points of every two adjacent lyrics detection frames in the sequence of lyrics detection frames. For example, the difference is calculated between the ordinates of the points at the upper left corners of the detection frames of the lyrics "I" and "sea". If the calculated difference is less than a preset percentage (such as 50%) of the average height of all the lyrics detection frames, it is determined that the detection frames of the lyrics "I" and "sea" are in the same lyrics line; the difference is calculated between the ordinates of the points at the upper left corners of the detection frames of the lyrics "I" and "eye", and if the calculated difference is greater than the preset percentage of the average height of all the lyrics detection frames, and the lyrics detection frame with the smaller ordinate of the point at the upper left corner is the detection frame of the lyrics "eye", it is determined that the detection frame of the lyrics "eye" is in the next line of the detection frame of the lyrics "I".

[0080] Another preferred implementation for correcting the detection frames is that two English lyrics detection frames in the target musical score with an overlapping area ratio exceeding a preset ratio can be merged into one English lyrics detection frame. Moreover, the situation where the detection frame of the English lyrics in a certain line is selected into other lines can be corrected. Specifically, for each line of English lyrics detection frames, the ordinate of the reference point closest to the head of the musical score among all the English lyrics detection frames in this line is used as the benchmark for this line, and based on this benchmark, the English lyrics detection frames in this line or the previous line are divided into an upper detection frame and a lower detection frame along the reading direction of the target musical score; if the height of the lower detection frame is less than a preset percentage (such as 50%) of the average height of all the English lyrics detection frames, the lower detection frame is deleted, and the upper detection frame is used as the new English lyrics detection frame; if the height of the upper detection frame is less than the preset percentage of the average height, the upper detection frame is deleted, and the lower detection frame is used as the new English lyrics detection frame.

[0081] As Figure 6 shown in a section of an English score, the area surrounded by the detection frame of the lyrics "Re-mem-ber" is too large, resulting in an overlap with the detection frame of the lyrics "(Im)ag-ine". At this time, assuming that the detection frame of the lyrics "(Im)ag-ine" is the detection frame with the point at the upper left corner closest to the head of the musical score in its line, the ordinate of the point at the upper left corner of this detection frame can be used as the benchmark for this line, and along the line parallel to the horizontal axis where this benchmark is located, the detection frame of the lyrics "Re-mem-ber" is divided, and the result can be as Figure 7If the detection frame shown on the left side of the arrow includes an upper detection frame and a lower detection frame, and the height of the lower detection frame is less than 50% of the average height, the lower detection frame is deleted; and the upper detection frame is retained, i.e. the detection frame shown on the right side of the arrow is obtained, which is taken as the new detection frame of the lyrics "Re-mem-ber", so that the detection frame of the lyrics is corrected to avoid overlapping with the detection frame of the lyrics in the next line.

[0082] In another case, if the lyrics "Re-mem-ber" and the lyrics "(Im)ag-ine" are in the same detection frame, the detection frame is also divided into an upper detection frame and a lower detection frame in the above manner, wherein the height of the upper detection frame is the average height, and the height of the lower detection frame is greater than 50% of the average height, and the lower detection frame encloses the lyrics "(Im)ag-ine", the lower detection frame is taken as the detection frame of the lyrics in the next line.

[0083] In addition, other ways of correcting the detection frame can be to correct the lyrics line detection result to correct the information misrecognized as other categories to lyrics; or to correct the head information detection frame to correct the detection frames overlapping with each other; or to correct the note detection result to delete the detection result overlapping by 50% of the area. The embodiment is not limited to the way of correcting the detection frame.

[0084] After the position information of each text element in the target music score is determined, the semantic information of each text element can be extracted according to the position information of each text element, which can be realized by using a text recognition algorithm, for example, the semantic information in the detection frame of each text element can be extracted by using the mature MASTER scheme in the OCR (optical character recognition) character recognition technology.

[0085] In a preferred manner, different text recognition models can be used to extract the semantic information of each text element according to different categories of text elements, for example, two text recognition models with the same network structure are used, one of which is used to extract the semantic information of the music head information, and the other of which is used to extract the semantic information of other text elements except the music head information. The network structures of the two models are the same, except that the network weights in the models are different.

[0086] 304. adjusting the overall structured information of the target music score according to the lyrics time information, so that the lyrics time information matches the music measure structure of the target music score;

[0087] In the embodiment, each element of the accompaniment score is determined according to the position information of each element in the target score. In a preferred embodiment, the bar line of the accompaniment score is displayed according to the position information of the bar line in the target score to determine each bar of the accompaniment score, the position of the head information in the accompaniment score is displayed according to the position information of the head information in the target score, the staff area of the accompaniment score is displayed according to the position information of the staff in the target score, the position area of the notes in the accompaniment score is displayed according to the position information of the notes in the target score, and so on.

[0088] After the position and area of each text element in the target score are determined, the semantic information of the notes in the bar of the accompaniment score is displayed according to the position information of the notes in the target score, the semantic information of the lyrics in the bar of the accompaniment score is displayed according to the position information of the lyrics in the target score, and the semantic information of the chord in the accompaniment score is displayed according to the position information of the chord in the target score, the semantic information of the head information in the accompaniment score is displayed according to the position information of the head information in the target score.

[0089] For the head information, the semantic information such as the beat, bpm, and mode of the head information in the target score is extracted according to the position information of the head information, and then the elements such as the notes, the tied notes, and the dots are distributed to the bars according to the position information of the bar line and the notes, and the lyrics are distributed to the bars according to the position information of the lyrics to generate the bars of the accompaniment score.

[0090] For example, if the vertical coordinate of the detection frame of a certain sentence of lyrics is between the vertical coordinate of the first bar of the first line and the vertical coordinate of the first bar of the second line, and the lyrics are generally located below the bar, it is determined that the sentence of lyrics corresponds to the first bar of the first line. If the first bar of the first line includes multiple bars, and the horizontal coordinate range of the detection frame of the lyrics is within the horizontal coordinate range of a certain bar, or most of the horizontal coordinate range of the lyrics is within the horizontal coordinate range of a certain bar, it is determined that the sentence of lyrics corresponds to the certain bar in the first bar of the first line.

[0091] For example, as shown in the score in Figure 5 For example, as shown in the score in

[0092] In this embodiment, when generating the accompaniment score, the semantic information of the lyrics corresponding to the notes in the measure can also be displayed according to the correspondence between the lyrics and the notes. Specifically, the first horizontal coordinate range of the detection box of each lyric of the target score is determined, and the second horizontal coordinate range of the detection box of each note in the measure where the lyric is located is determined, the intersection between the first horizontal coordinate range and each second horizontal coordinate range is determined respectively, the lyric and the note corresponding to the first horizontal coordinate range and the second horizontal coordinate range with the largest intersection range are determined to have a corresponding relationship, and the semantic information of the lyric having the corresponding relationship with the notes in the measure is displayed in the measure of the accompaniment score.

[0093] For example, the horizontal coordinate range of the first note detection box in the measure is (0, 20), the horizontal coordinate range of the second note detection box is (30, 50), and the horizontal coordinate range of the detection box of a certain lyric in the measure is (25, 50). Since the intersection range of the horizontal coordinate range of the second note detection box is the largest, it is determined that the lyric corresponds to the second note. Furthermore, in the accompaniment score, the semantic information of the lyric can be displayed below the second note, indicating the corresponding relationship with the second note.

[0094] For the repeat mark in the target score, when generating the accompaniment score, in order to facilitate the user to read the score, the second passage of the score represented by the repeat mark can be directly placed after the first passage of the score. In this way, after the user reads the first passage of the score, the user can directly read the second passage of the score from the end of the first passage of the score, without returning to the beginning of the first passage of the score to read the second passage of the score again, thereby facilitating the user to read the score.

[0095] As shown in Figure 8 , the score of a certain passage of a song is the score picture shown on the left side of the figure. Based on the method of this embodiment, the score picture of this passage can be converted into the accompaniment score of the guitar, that is, converted into the guitar tablature as shown on the right side of the figure on the smart terminal.

[0096] The target score only has the function of display and cannot support the user to edit the elements in the target score, such as the user cannot edit the chord, the pitch / key signature, etc. in the picture. Therefore, the target score needs to be processed based on the method of this embodiment.

[0097] In this embodiment, the elements of the score picture include text elements and non-text elements. The text elements include text content, such as elements containing text content such as chords, lyrics, notes, header information, etc., and the non-text elements do not include text content, such as elements not containing text content such as bar lines, dots, and ties.

[0098] The semantic information of the non-text element has been determined in the detection process of the target music score element detection model, that is, the label information of the non-text element represents the semantic information. The semantic information of the text element includes not only the label information representing the category, but also the specific content, such as the semantic information of the lyrics, which includes the specific lyrics content, and the semantic information of the chord, which includes the information representing which chord it belongs to. Therefore, in this embodiment, the semantic information of each text element in the target music score is also extracted, that is, the semantic information of each text element is extracted according to the position information of each text element in the target music score, so as to generate an editable music score according to the semantic information of the text element.

[0099] The computer device can determine each element of the accompaniment score according to the position information of each element in the target music score. For example, the bar lines in the target music score divide the music score into multiple bars, and therefore, according to the positions of the multiple bar lines, the multiple bars of the accompaniment score are displayed. If there are tie notes, dots or rests between the notes in the target music score, the accompaniment score can display the tie notes according to the position information of the tie notes in the target music score, display the dots according to the position information of the dots in the target music score, and display the rests according to the position information of the rests in the target music score, and so on.

[0100] The semantic information of the text element in the target music score also needs to be displayed in the accompaniment score. Therefore, after determining the multiple bars of the accompaniment score, the semantic information of each text element is displayed at the position of the text element in the accompaniment score, such as the text elements of notes, lyrics, chords of the music score, and header information of the music score, so as to generate a complete accompaniment score. Therefore, the computer device can display the accompaniment score, and the user can input editing operations in the accompaniment score to change the elements therein, such as changing the pitch of the music score, transposing the music score, or changing the chords, and so on.

[0101] In this embodiment, in addition to the editing of the accompaniment score by the user, the music score segment corresponding to the currently playing music segment can also be displayed according to the playing progress of the music, and the music score segment includes the above-mentioned overall structured information based on the target music score. Specifically, the lyrics in the accompaniment score can be lyrics in QRC format, and the lyrics in QRC format carry the timestamps of each lyric. Since the positions of the notes and the lyrics in the accompaniment score correspond to each other, when the music is played, the lyrics corresponding to the time of the currently playing music segment can be displayed according to the timestamps, and the music score segment in which the lyrics are located, that is, the elements including the lyrics and the notes, the staff, and the like.

[0102] The AI music score processing method in the embodiments of the present application is described above, and the computer device in the embodiments of the present application is described below. Please refer to Figure 9 An embodiment of the computer device in the embodiments of the present application includes:

[0103] The computer device 900 can include one or more central processing units (CPU) 901 and a memory 905 in which one or more applications or data are stored.

[0104] The memory 905 can be volatile memory or persistent memory. The programs stored in the memory 905 can include one or more modules, each of which can include a series of instruction operations in the computer device. Further, the central processing unit 901 can be configured to communicate with the memory 905 to execute the series of instruction operations in the memory 905 on the computer device 900.

[0105] The computer device 900 can also include one or more power supplies 902, one or more wired or wireless network interfaces 903, one or more input / output interfaces 904, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0106] The central processing unit 901 can execute the operations of the computer device in the above-described Figures 2 to 3 embodiments, which will not be described here in detail.

[0107] The embodiments of the present application also provide a computer storage medium, one of which includes: the computer storage medium stores instructions, and the instructions, when executed on a computer, cause the computer to perform the operations of the computer device in the above-described Figures 2 to 3 embodiments.

[0108] The embodiments of the present application also provide a computer program product, one of which includes: the computer program product, when running on a computer device, causes the computer device to perform the operations of the computer device in the above-described Figures 2 to 3 embodiments.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the above-described method embodiments, which will not be described here in detail.

[0110] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0111] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0112] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0113] When the integrated unit is implemented in the form of software functional units and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or partly, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk, and various other media that can store program codes.

Claims

1. An AI score processing method, characterized by, The method comprises: inputting a target music score to be processed into a pre-trained target music score element detection model to obtain a plurality of elements of the target music score and label information of the plurality of elements; obtaining overall structured information of the target music score according to the plurality of elements and the label information of the plurality of elements, the overall structured information of the target music score being a music bar structure of the target music score; obtaining lyrics information of the target music score, the lyrics information comprising lyrics text and lyrics time information; adjusting the overall structured information of the target music score according to the lyrics time information, so that the lyrics time information matches the music bar structure of the target music score; wherein the semantic information of the lyrics is displayed in a bar of an accompaniment score according to position information of the lyrics in the target music score, comprising: determining a first horizontal coordinate range of a detection box of each lyrics of the target music score, and determining a second horizontal coordinate range of a detection box of each note in a bar in which each lyrics is located; determining an intersection between the first horizontal coordinate range and each second horizontal coordinate range, respectively, and determining that the lyrics and the note corresponding to the first horizontal coordinate range and the second horizontal coordinate range, respectively, have a corresponding relationship, the intersection being the largest; displaying the semantic information of the lyrics having the corresponding relationship with the note in the bar in the bar of the accompaniment score.

2. The method of claim 1, wherein, The obtaining of the overall structured information of the target music score according to the plurality of elements and the label information of the plurality of elements comprises: allocating lyrics, chords and sustain to a bar of an accompaniment score according to position distribution information of bar lines and notes in the plurality of elements; processing missing or incorrect bars in the accompaniment score according to song beat information, eliminating misrecognized data in the bars according to position information of the elements, and processing a corresponding relationship between lyrics and notes in a beat; processing repeated bars in combination with repeated identification and multiple lines of lyrics in a bar of the target music score to obtain the overall structured information of the target music score.

3. The method of claim 2, wherein, The processing of the repeated bars in combination with the repeated identification and the multiple lines of lyrics in the bar of the target music score comprises: matching lyrics time information of the multiple lines of lyrics in the bar of the target music score with time information corresponding to the bar of the target music score to determine position information of the lyrics; adjusting a corresponding relationship between the bar of the accompaniment score and the lyrics in the bar of the accompaniment score based on the repeated identification to obtain the overall structured information of the accompaniment score based on the target music score.

4. The method of claim 1, wherein, The elements comprise text elements, the text elements comprising lyrics, and a detection box of the text elements comprising a lyrics detection box; after obtaining the plurality of elements of the target music score output by the target music score element detection model, the method further comprises: determining a point with a maximum vertical coordinate and a point with a minimum vertical coordinate in each lyrics detection box in a coordinate system determined based on the target music score, taking a difference between the maximum vertical coordinate and the minimum vertical coordinate as a height of the lyrics detection box, and counting an average height of all lyrics detection boxes of the target music score; Sort the plurality of lyric bounding boxes in ascending or descending order of the vertical coordinates of the reference points, to obtain a sequence of lyric bounding boxes; the reference point is any point in the lyric bounding box; Determine that the two lyric bounding boxes whose vertical coordinates are subtracted are in the same row if the difference is less than or equal to the preset percentage of the average height; If the difference is greater than the preset percentage of the average height, take the lyric bounding box with the smaller vertical coordinate of the reference point as the lyric bounding box of the next row.

5. The method of claim 4, wherein, The lyrics are English lyrics, and the lyric bounding boxes are English lyric bounding boxes; After obtaining the plurality of elements of the target musical score output by the target musical score element detection model, the method further comprises: For each row of English lyric bounding boxes, take the vertical coordinate of the reference point closest to the musical score header in all English lyric bounding boxes in the row as the reference of the row, and divide the English lyric bounding boxes in the row or the previous row into upper bounding boxes and lower bounding boxes along the reading direction of the target musical score according to the reference; If the height of the lower bounding box is less than the preset percentage of the average height, delete the lower bounding box and take the upper bounding box as a new English lyric bounding box; If the height of the upper bounding box is less than the preset percentage of the average height, delete the upper bounding box and take the lower bounding box as a new English lyric bounding box.

6. The method of claim 1, wherein, The elements include text elements and non-text elements; the non-text elements include bar lines; and the text elements include notes, lyrics, chords, and musical score header information; The method further comprises: Display the bar lines of the accompaniment score according to the position information of the bar lines in the target musical score to determine each bar of the accompaniment score; Display the semantic information of the notes in the bars of the accompaniment score according to the position information of the notes in the target musical score, and display the semantic information of the lyrics in the bars of the accompaniment score according to the position information of the lyrics in the target musical score; Display the semantic information of the chords in the accompaniment score according to the position information of the chords in the target musical score, and display the semantic information of the musical score header information in the accompaniment score according to the position information of the musical score header information in the target musical score.

7. The method of claim 1, wherein, After adjusting the overall structured information of the target musical score according to the lyric time information to obtain the accompaniment score corresponding to the target musical score, the method further comprises: Display the musical score segment corresponding to the currently playing music segment in the accompaniment score according to the playing progress of the music, wherein the musical score segment comprises the overall structured information based on the target musical score.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 7.

9. A computer storage medium, characterized in that The computer storage medium stores instructions, and the instructions make the computer execute the method of any one of claims 1 to 7 when executed on the computer.

Images