A mediumless holographic device for musical instruments and its control method

Abstract

An embodiment of the present invention provides a mediumless holographic device for musical instruments and a control method thereof, which relates to the technical field of mediumless display technology for musical score information. The mediumless holographic device for musical instruments includes: a mediumless holographic module for generating and displaying a mediumless holographic musical score image; a first sensor for collecting musical instrument motion information and / or musical instrument sound information during the operation of the musical instrument; and a control module signal-connected to the first sensor and the mediumless holographic module for determining musical score information according to the musical instrument motion information and / or the musical instrument sound information, and adjusting the mediumless holographic musical score image according to the musical score information. By means of the present invention, the problem of high complexity of musical score operation is solved, and thus the effects of improving the performance effect and reducing performance mistakes are achieved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of medium - less display of musical score information, and more specifically, to a medium - less holographic device for musical instruments and its control method. Background Art

[0002] In the field of musical instrument performance, musical scores are an important basis for performers. In traditional musical instrument performances, performers usually need to manually flip through paper musical scores during the performance. This method not only easily distracts the performer's attention, affecting the coherence and accuracy of the performance, but also may bring greater psychological pressure to the performer during complex movements or fast performances. In addition, for some large - scale performances or professional performance occasions, special staff are usually required to assist in page - turning, which not only increases the labor cost, but also may cause the performance to be interrupted due to operation errors, affecting the performance effect.

[0003] With the development of technology, electronic musical score display devices have gradually been introduced into musical instrument performances. These devices display musical scores through a display screen, and can achieve automatic page - turning and adjustment of musical scores, solving the deficiencies of traditional paper musical scores to a certain extent. However, most of the existing electronic musical score display devices rely on touch - screen operations or external control devices. Performers still need to be distracted to operate during the performance, and the devices usually need to maintain a certain physical connection with the musical instrument, restricting the performer's activity range and the flexibility of the performance.

[0004] In response to the above problems, there is currently no good solution. Summary of the Invention

[0005] Embodiments of the present invention provide a medium - less holographic device for musical instruments and its control method to at least solve the problem of high complexity of musical score operations in related technologies.

[0006] According to an embodiment of the present invention, there is provided a medium - less holographic device for musical instruments, including:

[0007] A medium - less holographic module for generating and displaying a medium - less holographic musical score image;

[0008] A first sensor for collecting musical instrument movement information and / or musical instrument sound information during the operation of the musical instrument, where the musical instrument movement information includes vibration actions or pressed / pulled actions performed by the keys or strings of the musical instrument during operation, and the musical instrument sound information includes information about the sound emitted by the musical instrument;

[0009] A control module, signal - connected to the first sensor and the medium - less holographic module, for determining musical score information according to the musical instrument movement information and / or the musical instrument sound information, and adjusting the medium - less holographic musical score image according to the musical score information.

[0010] In an exemplary embodiment, it further includes:

[0011] A second sensor, which is signal - connected to the control module and is used to collect external action information of external actions operating on the musical instrument;

[0012] The control module determines score information according to the external action information, the musical instrument action information, and / or the musical instrument sound information, and adjusts the medium - less holographic score image according to the score information.

[0013] In an exemplary embodiment, the medium - less holographic module includes:

[0014] A display, which is signal - connected to the control module and is used to generate a score image according to the instruction of the control module;

[0015] An optical waveguide plate, which is used to perform optical processing on the score image to generate a medium - less holographic score image.

[0016] In an exemplary embodiment, the medium - less holographic device for musical instruments is connected to the body of the musical instrument;

[0017] Or,

[0018] The medium - less holographic device for musical instruments is not connected to the body of the musical instrument.

[0019] According to another embodiment of the present invention, a method for controlling a medium - less holographic device for musical instruments is provided, including:

[0020] Obtaining musical instrument action information and / or musical instrument sound information of the operation of the musical instrument through a first sensor, where the musical instrument action information includes vibration actions or pressed / pulled actions performed by the keys or strings of the musical instrument when the musical instrument is operating, and the musical instrument sound information includes information about the sound emitted by the musical instrument;

[0021] The control module determines score information according to the musical instrument action information and / or the musical instrument sound information, and the score information is used to indicate the score section corresponding to the musical instrument action information or the musical instrument sound information;

[0022] The medium - less holographic module adjusts the pre - generated medium - less holographic score image according to the score information.

[0023] In an exemplary embodiment, it further includes:

[0024] Collecting external action information through a second sensor, where the external action information includes external action information of external actions operating on the musical instrument;

[0025] The control module determines the sheet music information based on the external action information, the musical instrument action information, and / or the musical instrument sound information, and adjusts the mediumless holographic sheet music image according to the sheet music information.

[0026] In an exemplary embodiment, the determining the sheet music information according to the musical instrument action information includes:

[0027] Determine the syllable information corresponding to the musical instrument action according to the musical instrument action information;

[0028] Match the syllable information with a preset sheet music to determine the sheet music information.

[0029] In an exemplary embodiment, the determining the sheet music information according to the musical instrument sound information includes:

[0030] Determine the syllable information corresponding to the musical instrument sound according to the musical instrument sound information;

[0031] Match the syllable information with a preset sheet music to determine the sheet music information.

[0032] According to another embodiment of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and wherein the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0033] According to another embodiment of the present invention, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0034] Through the present invention, since the musical instrument sound or the musical instrument action is collected and recognized, automatic sheet music recognition and page turning can be performed without manual operation. Therefore, the problem of high complexity of sheet music operation can be solved, and the effect of improving the musical instrument performance effect and reducing the musical instrument performance difficulty can be achieved. Description of the Drawings

[0035] Figure 1 is a structural block diagram of a mediumless holographic device for musical instruments according to an embodiment of the present invention;

[0036] Figure 2 is a schematic principle according to a specific embodiment of the present invention Figure 1 ;

[0037] Figure 3 is a schematic principle according to a specific embodiment of the present invention Figure 2 ;

[0038] Figure 4Schematic illustration according to the principles of specific embodiments of the present invention Figure 3 ;

[0039] Figure 5 It is a flowchart of a control method for a mediumless holographic device for musical instruments according to an embodiment of the present invention. Specific embodiments

[0040] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0041] Hereinafter, terms such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0042] In addition, in the present application, orientation terms such as "upper", "lower", "left", and "right" may include, but are not limited to, being defined relative to the schematic placement of components in the accompanying drawings. It should be understood that these directional terms may be relative concepts, which are used for relative description and clarification, and may change accordingly with the change of the orientation of the components in the accompanying drawings.

[0043] In the present application, unless otherwise clearly defined and limited, the term "connection" should be understood in a broad sense. For example, "connection" may be a fixed connection, a detachable connection, or an integral body; it may be directly connected or indirectly connected through an intermediate medium. In addition, the term "coupling" may be a way of electrical connection for signal transmission.

[0044] As used herein, "about", "substantially" or "approximately" includes the stated value and the average value within an acceptable deviation range of the specific value, where the acceptable deviation range is determined by those of ordinary skill in the art considering the measurement being discussed and the errors associated with the measurement of the specific quantity (i.e., the limitations of the measurement system).

[0045] In this embodiment, a mediumless holographic device for musical instruments is also provided. As used hereinafter, the term "module" may be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0046] In general, when a musical instrument player is performing, they need to turn the pages of the sheet music. This method places high requirements on the player for some complex movements or movements that need to be processed quickly. If the display for the sheet music is remotely operated by staff, it increases the labor cost and is prone to operation errors. Therefore, a new sheet music display method is needed to adapt to this performance scenario.

[0047] Embodiment 1

[0048] Figure 1 is a structural block diagram of a mediumless holographic device 1 for musical instruments according to an embodiment of the present invention, as Figure 1 shown. The mediumless holographic device 1 for musical instruments includes:

[0049] A mediumless holographic module 11, a first sensor 12, and a control module 13. The first sensor 12 is used to collect musical instrument movement information and / or musical instrument sound information during the operation of the musical instrument. The musical instrument movement information includes the actions performed by the keys or strings of the musical instrument 2 during operation, such as the vibration of the keys or strings of the musical instrument 2 itself or the actions of pressing or pulling the keys or strings of the musical instrument. The musical instrument sound information includes the information of the sound emitted by the musical instrument 2. The control module 13 is signal-connected to the first sensor 12 and the mediumless holographic module 11. The mediumless holographic module 11 is used to generate and display a mediumless holographic sheet music image 113 (as Figure 2 and Figure 3 shown). The mediumless holographic sheet music image 113 can be a staff notation or other types of sheet music (such as dynamic operation prompts for indicating actions or keys). The control module 13 determines the sheet music information based on the musical instrument movement information and / or the musical instrument sound information, and adjusts the mediumless holographic sheet music image 113 according to the sheet music information. For example, when the first sensor 12 collects a segment of sound or action, and the control module 13 then identifies that the sound or action corresponds to a certain movement in the sheet music, it then instructs the mediumless holographic module 11 to turn the page of the sheet music or make a change to the movement, so that the player can perform the movement without having to turn the page themselves. Specifically, during the process of turning the page or changing the movement, prompts for the current action or syllable can also be provided according to the sound or action to indicate the current syllable position and ensure the correctness of the performance. Of course, the prompting method can be a triangular indication mark, special color display, etc. There is no limitation here as long as it can provide a prompt and falls within this scope.

[0050] Among them, the musical instrument 2 can be a keyboard musical instrument with certain keys such as a piano or an electronic organ, or a wind instrument such as a saxophone, a suona horn, or a wind pipe, a string instrument such as an erhu, a violin, or a harp, a percussion instrument such as a dulcimer or a drum, that is, idiophones, membranophones, aerophones, chordophones, electrophones, etc. are all within the scope of the musical instrument referred to in this application; the first sensor 12 can be a combination of an array microphone and a sound sensor for sound collection, or a high-pixel camera or a light sensor for motion collection, or other devices capable of collecting sound or motion, such as a combination of a radar, a camera, and a sound sensor, a combination of an infrared sensor, a camera, and a sound sensor, a combination of a radar, a camera, an infrared sensor, and a sound sensor, etc., which is not limited here; the control module 13 can be a CPU / GPU combination for image and sound processing; the motion collected here includes the motion of the musical instrument itself, such as the motion change of the vibration of the keys / strings themselves when the keys are pressed or the strings are plucked or pulled.

[0051] Of course, to recognize actions or sounds, it is necessary to utilize a pre-trained neural network model or other deep learning models. Specifically, it includes first performing denoising processing on the collected sound information and / or action information to remove background noise and retain clear instrument sounds or instrument actions. Subsequently, the data is normalized to standardize the data to the same range for better model processing. Then, useful features are extracted from the original data. For example, for sound data, spectral features (such as MFCC, Mel Frequency Cepstral Coefficients) can be extracted, while for action data, the motion trajectories of key points can be extracted. Subsequently, this data is input into the corresponding model. For example, a Convolutional Neural Network (CNN) model suitable for processing image and video data, which can be used to recognize instrument actions, or a Recurrent Neural Network (RNN) model suitable for processing time series data, which can recognize sound signals. Of course, relevant models based on the Transformer architecture can also be used, which is not limited here. After that, the model automatically extracts features and makes predictions, outputs the corresponding musical score information or syllable information, and matches the recognized syllable information or musical score information with a preset musical score. The preset musical score at this time can be stored in a database. Then, through a search algorithm (such as the Dynamic Time Warping (DTW) algorithm), the recognized syllable sequence is compared with the syllable sequence in the musical score to find the most matching musical score section. Subsequently, according to the matched musical score information, an instruction to adjust the mediumless holographic musical score image is generated. For example, if it is recognized that the current movement being played requires turning the page, a page-turning instruction is generated; if it is necessary to prompt the current syllable position, a syllable prompt instruction is generated. Then, the adjustment instruction is transmitted to the mediumless holographic module to update the content of the mediumless holographic musical score image by controlling the light waveguide plate and the display device, so that it is consistent with the playing progress of the performer. Among them, during the model optimization process, if the matching is successful, the position of the musical score being played currently is determined; if the matching fails, a prompt message is issued to remind the performer or the system to perform further processing, such as further optimizing and updating the model according to the output result and the matching result, etc.

[0052] As Figure 2 shown, the mediumless holographic module 11 includes a display device 111 and a light waveguide plate 112. Among them, the display device 111 is signal-connected to the control module 13 and is used to generate a mediumless holographic musical score image 113 according to the instruction of the control module. The display device 111 can be a display screen, a display, or other display devices or equipment that can generate a light source image. The light waveguide plate 112 is used to perform optical processing on the musical score image to generate a mediumless holographic musical score image 113. The light waveguide plate 112 can be the optical element or equipment described in any of the Chinese patents with application numbers 202210060077.2, 202221492951.1, and 201920104395.8, or other optical elements or equipment that can perform floating imaging, which is not limited here.

[0053] It should be noted that due to the different types of musical instrument 2, the mediumless holographic device 1 for musical instruments can be connected to the body of the musical instrument 2 or not connected to the body of the musical instrument 2. The connection here can be a direct connection such as a fixed connection or a detachable connection, or a signal connection through a cable or a communication device. Among them, the fixed connection includes welding, gluing, and a connection method that is not easily disassembled after connection through a specific installation structure. The detachable connection includes bolt fixation, Velcro, etc. that can be disassembled and assembled; the signal connection can be to install the first sensor 12 on the body of the musical instrument 2, and other modules are installed in other positions, and the signal is transmitted through a signal protocol or a cable, which is not limited here; it is easy to understand that the non-connection here means that the mediumless holographic device 1 for musical instruments does not have direct contact with the body of the musical instrument 2, but directly collects the sound or motion information of the musical instrument 2, and the sound or motion information is directly recognized after being screened; the body of the musical instrument 2 refers to the part that can perform musical instrument playing, such as the body of a guitar, a piano, a violin, and the body of a wind instrument, etc.; among them, if it is a musical instrument such as a piano with a certain installation space, the mediumless holographic device 1 for musical instruments can be installed inside the body of the piano. If it is a musical instrument such as a guitar without an installation space, the mediumless holographic device 1 for musical instruments is not connected to the body of the musical instrument 2, specifically as Figure 3 - Figure 4 shown Figure 3 In (A) structure in, the mediumless holographic device 1 for musical instruments is externally placed outside the body of the musical instrument 2 and is connected to the body of the musical instrument 2; Figure 3 In (B) structure in, the mediumless holographic device 1 for musical instruments is externally placed outside the body of the musical instrument and is not connected to the body of the musical instrument 2; Figure 4 The structure in is that the mediumless holographic device 1 for musical instruments is externally placed outside the body of the musical instrument 2, but can be built-in or pushed and pulled by means of hinging; Figure 4 The structure in is that the mediumless holographic device 1 for musical instruments is built-in inside the body of the musical instrument 2 and is not hinged or pushed and pulled.

[0054] In particular, due to different installation positions and usage requirements, the image display method and usage method of the mediumless holographic device for musical instruments are also different. Taking a piano as an example:

[0055] As Figure 4 shown in the (B) structure in, it can be that the mediumless holographic device 1 for musical instruments is fixedly built-in in the body of the piano. At this time, the generated mediumless music score image can be located above the body of the piano or above the keys (i.e., in front of the body of the piano), or it can be installed on the upper surface of the body of the piano. Here, the generated mediumless music score image can be located above the body of the piano or above the keys (i.e., in front of the body of the piano). It can also be that the mediumless holographic device 1 for musical instruments is built-in in the body of the piano, and at the same time, the mediumless holographic device 1 is hinged to the body of the piano (such asFigure 4 As shown in the structure of (A), the dashed part is in a hinged, rotating, pushing and pulling state. Thus, the performer can push, pull, or flip the mediumless holographic device for musical instruments according to their own needs, thereby changing the display position or display angle of the mediumless sheet music image. The specific hinged or installation method is not limited here as long as the above functions can be achieved. Of course, in addition to hinging, other methods that can change the position of the built-in mediumless holographic device 1 for musical instruments to change the display position of the mediumless holographic sheet music image 113 also fall within the protection scope of this application. For example, methods of changing the position of the mediumless holographic device 1 for musical instruments through special sliding structures, fixedly connected electric push rods, cylinders or other devices or structures are all within the above scope and are not limited here.

[0056] Embodiment 2

[0057] The difference from Embodiment 1 is that, to further accurately identify the movement situation, the movement situation of the performer can also be collected, and the progress of the movement can be identified according to the movement of the performer. Specifically, the mediumless holographic device for musical instruments further includes:

[0058] A second sensor 14, which is signal-connected to the control module 13 and is used to collect external action information on the operation of the musical instrument by an external action.

[0059] The control module 13 determines the sheet music information according to the external action information, the musical instrument action information, and / or the musical instrument sound information, and adjusts the mediumless holographic sheet music image 113 according to the sheet music information.

[0060] Generally, in a fixed movement, the actions of the performer on the musical instrument are fixed. Whether it is hitting, blowing, pressing, pulling, etc., it is necessary to maintain a fixed position to achieve the precise performance of the movement. Therefore, at this time, by capturing and identifying the movement changes and their movement distributions of the performer within a period of time, the current movement situation can be determined, and thus it can be judged whether it is necessary to turn the page or change the movement, etc.

[0061] It is easy to understand that the second sensor 14 can also be a sensor capable of capturing the movement of the performer, such as at least any one of an infrared sensor, a light sensor, a camera, a radar, etc., which is not limited here; and the installation position and installation method of the second sensor 14 are preferably based on being able to capture the limb movements of the performer, such as the movements of limb parts such as fingers, elbows, and arms. And for more accurate identification, multiple sensors need to be set at multiple positions; the external action information is the limb actions performed by the performer on the musical instrument, such as hitting, blowing, pressing, pulling, etc.

[0062] It should be noted that the above-mentioned modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited thereto: the above-mentioned modules are all located in the same processor; or, the above-mentioned modules are respectively located in different processors in any combination form.

[0063] Embodiment 3

[0064] In this embodiment, a method for controlling a mediumless holographic device for musical instruments is provided. Figure 5 It is a flowchart of a method for controlling a mediumless holographic device for musical instruments according to an embodiment of the present invention, as Figure 5 shown, this process includes the following steps:

[0065] Step S51, obtain musical instrument action information and / or musical instrument sound information of the working musical instrument through the first sensor 12, wherein the musical instrument action information includes vibration actions or pressed / pulled actions performed by the keys or strings of the musical instrument when the musical instrument is working, and the musical instrument sound information includes information on the sound emitted by the musical instrument;

[0066] Step S52, the control module 13 determines musical score information according to the musical instrument action information and / or musical instrument sound information, and the musical score information is used to indicate the musical score section corresponding to the musical instrument action information or the musical instrument sound information;

[0067] Step S53, the mediumless holographic module 11 adjusts the pre-generated mediumless holographic musical score image according to the musical score information.

[0068] Among them, the execution subject of the above steps can be a base station, a terminal, etc., but not limited thereto.

[0069] In an optional embodiment, this method further includes:

[0070] Step S54, collect external action information through the second sensor 14, wherein the external action information includes external action information of an external action operating on the musical instrument;

[0071] Step S55, the control module 13 determines musical score information according to the external action information, the musical instrument action information and / or the musical instrument sound information, and adjusts the mediumless holographic musical score image according to the musical score information.

[0072] Embodiment 4

[0073] Determining the musical score information according to the musical instrument action information includes:

[0074] Step S521, determine syllable information corresponding to the musical instrument action according to the musical instrument action information;

[0075] Step S522: Match the syllable information with a preset musical score to determine the musical score information.

[0076] In this embodiment, collecting the instrument actions is to judge the progress of the movement from the movement situation of the instrument itself, so as to effectively facilitate the movement progress prompt or automatic page turning for the performer.

[0077] Specifically, the collected instrument action information includes the actions of the instrument keys or strings. For example, the speed, strength, and duration of the key presses, or the amplitude and frequency of the string plucks; then analyze the collected action information to extract the feature parameters related to the syllables. For example, for keyboard instruments, the played notes can be inferred according to the order, speed, and duration of the key presses; for string instruments, the pitch and intensity can be determined according to the frequency and amplitude of the string plucks; for guitar playing, the corresponding notes can be determined by detecting the plucking action and the fret position; then according to the parsed feature parameters, determine the syllable information corresponding to the instrument actions. For example, by analyzing the order and time interval of the key presses, the sequence of played notes can be determined; by analyzing the frequency of the string plucks, the pitch can be determined, and then the syllable can be determined; the syllable information can include parameters such as the pitch, duration, and intensity of the notes, and these parameters together constitute the syllable information corresponding to the instrument actions.

[0078] The preset musical score can be a standard musical score stored in the database or a user-defined musical score; the preset musical score information is stored in the system database; match the extracted syllable information with the preset musical score. The dynamic time warping (DTW) algorithm, the longest common subsequence (LCS) algorithm, or other similarity matching algorithms can be used to compare the similarity between the syllable information and the musical score. During the matching process, the system will compare the parameters such as the pitch, duration, and intensity of the syllable information with the notes in the musical score one by one to find the most matching part; the determined musical score information includes parameters such as the pitch, duration, and intensity of the notes, as well as the arrangement order and movement structure of the notes. The musical score information can be stored in digital format, such as using the MusicXML format or the MIDI format, to facilitate the system to read and match; by matching the syllable information with the preset musical score, the musical score information of the current performance can be determined, and the matching process can include the comparison of aspects such as the duration, pitch, and rhythm of the notes to ensure the accuracy of the generated musical score information.

[0079] Embodiment 5

[0080] The difference from Embodiment 4 is that in order to better ensure the accuracy of data matching, the collected instrument action information can be further analyzed to judge the movement progress situation by analyzing whether the instrument action information is normal.

[0081] Specifically, the collected musical instrument motion information (such as key press speed, intensity, duration, motion frequency, motion amplitude, etc.) can be normalized. Subsequently, the normalized musical instrument motion information is constructed into a musical instrument motion matrix. Then, the correlation value of the musical instrument motion matrix is calculated according to the Pearson coefficient. If the correlation value is within a preset range, it is preliminarily determined that the relevant motion is reasonable. Subsequently, according to the musical instrument motion matrix, the matrix corresponding to the preset musical score with the highest similarity value to the musical instrument motion matrix is matched, and the preset musical score corresponding to this matrix is used as the optional musical score information. If the subsequent actual performance musical score change is the same as the optional musical score information, the optional musical score information is used as the final musical score information, and so on.

[0082] Embodiment 6

[0083] In an alternative embodiment, the determining the musical score information according to the musical instrument sound information includes:

[0084] Step S523, determining the syllable information corresponding to the musical instrument sound according to the musical instrument sound information;

[0085] Step S524, matching the syllable information with a preset musical score to determine the musical score information.

[0086] In this embodiment, collecting the musical instrument motion is to judge the movement progress of the movement from the sound situation of the musical instrument, so as to effectively facilitate the movement progress prompt or automatic page turning for the performer.

[0087] Among them, the fundamental frequency of the sound signal can be extracted by methods such as autocorrelation algorithm and cepstrum analysis to determine the pitch of the note; the loudness of the note can be determined by analyzing the amplitude of the sound signal; the duration of the note can be determined by methods such as short-time energy analysis; the sound characteristics of different musical instruments can be distinguished by analyzing the spectral characteristics of the sound signal; subsequently, according to the extracted feature vectors, the sound signal is classified or regression analyzed by using a pre-trained sound recognition model (such as a deep learning model) to determine the syllable information corresponding to the musical instrument sound; for example, for the sound signal of a piano performance, the model can identify specific notes (such as C4, D4, etc.) and their durations according to the extracted features such as pitch, intensity, and duration. Among them, the syllable information can be expressed as a series of notes and their corresponding durations and intensities, for example: [C4, 1 / 4], [E4, 1 / 4], [G4, 1 / 2], indicating that the C4 note (duration 1 / 4 beat), E4 note (duration 1 / 4 beat), and G4 note (duration 1 / 2 beat) are played. Subsequently, according to the matching result display, the extracted syllable information matches the note sequence of the first measure and the first to third beats in the musical score. At this time, there is no need to turn the page, and so on.

[0088] Embodiment 7

[0089] The difference from Embodiment 6 is that in order to better ensure the accuracy of data matching, the collected musical instrument sound information can be further analyzed, and the progress of the movement can be judged by analyzing whether the musical instrument sound information is normal.

[0090] Specifically, the collected musical instrument motion information (such as fundamental frequency, loudness, amplitude, duration, spectral characteristics, etc.) can be normalized, and then the normalized musical instrument sound information is constructed into a musical instrument sound matrix. Then, the correlation value of the musical instrument sound matrix is calculated according to the Pearson coefficient. If the correlation value is within the preset range, it is preliminarily determined that the relevant sound is reasonable. Subsequently, according to this musical instrument sound matrix, the matrix corresponding to the preset musical score with the highest similarity value to this musical instrument sound matrix is matched, and the preset musical score corresponding to this matrix is used as the optional musical score information. Subsequently, if the change of the actual played musical score in the follow-up is the same as this optional musical score information, then this optional musical score information is used as the final musical score information, and so on.

[0091] Embodiment 8

[0092] The difference from Embodiment 5 and Embodiment 7 is that after obtaining the optional musical score information obtained according to the musical instrument motion matrix and the musical instrument sound matrix, the similarity between the two musical scores is calculated. When the similarity meets the requirements, the optional musical score information can be directly used as the final musical score information without comparing it with the actual playing situation; or the optional musical score information can be compared with the actual playing situation to ensure the accuracy of the performance.

[0093] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0094] An embodiment of the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is set to execute the steps in any one of the above method embodiments when running.

[0095] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0096] An embodiment of the present invention further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0097] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device. Among them, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0098] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0099] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in an electrical, mechanical or other form.

[0100] The units described as separate components may or may not be physically separated. The components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in the various embodiments of the present application may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0102] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0103] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A medium-free holographic device for musical instruments, characterized in that, Comprising: A mediumless holographic module for generating and displaying a mediumless holographic music score image; wherein, the mediumless holographic music score image is floating imaging; A first sensor for collecting instrument movement information and / or instrument sound information during the operation of the instrument, wherein the instrument movement information includes vibration actions or pressing / pulling actions performed by the keys or strings of the instrument during operation, and the instrument sound information includes information about the sound emitted by the instrument; A control module, signal-connected to the first sensor and the mediumless holographic module, for determining music score information based on the instrument movement information and / or the instrument sound information, and adjusting the mediumless holographic music score image according to the music score information; Among them, determining music score information based on the instrument movement information includes: normalizing the instrument movement information, wherein the instrument movement information includes key pressing speed, strength, duration, movement frequency, movement amplitude; constructing the normalized instrument movement information into an instrument movement matrix, calculating the correlation value of the instrument movement matrix according to the Pearson coefficient, if the correlation value is within a preset range, then preliminarily determining that the relevant action is reasonable; according to the instrument movement matrix, matching the first matrix corresponding to the preset music score with the highest similarity value to the instrument movement matrix, and taking the preset music score corresponding to the first matrix as the optional first music score information, if the change of the actually played music score is the same as the first music score information, then taking the first music score information as the final music score information; Determining music score information based on the instrument sound information includes: normalizing the instrument sound information; constructing the normalized instrument sound information into an instrument sound matrix, calculating the correlation value of the instrument sound matrix according to the Pearson coefficient, if the correlation value is within a preset range, then preliminarily determining that the relevant sound is reasonable; according to the instrument sound matrix, matching the second matrix corresponding to the preset music score with the highest similarity value to the instrument sound matrix, and taking the preset music score corresponding to the second matrix as the optional second music score information; if the change of the actually played music score is the same as the second music score information, then taking the second music score information as the final music score information.

2. The mediumless holographic device for musical instruments according to claim 1, wherein, Further comprising: A second sensor, signal-connected to the control module, for collecting external movement information of external actions operating on the instrument; The control module determines music score information based on the external movement information, the instrument movement information and / or the instrument sound information, and adjusts the mediumless holographic music score image according to the music score information.

3. The mediumless holographic device for musical instruments according to claim 1, wherein, The mediumless holographic module includes: A display component, signal-connected to the control module, for generating a music score image according to the instruction of the control module; An optical waveguide plate for optically processing the music score image to generate a mediumless holographic music score image.

4. The mediumless holographic device for musical instruments according to claim 1, wherein, The mediumless holographic device for the instrument is connected to the body of the instrument; Or, The mediumless holographic device for the instrument is not connected to the body of the instrument.

5. A control method for a medium-free holographic device for musical instruments, characterized in that, Comprising: Obtain instrument action information and / or instrument sound information of the instrument during operation through a first sensor, where the instrument action information includes the vibration actions or the pressed / pulled actions performed by the keys or strings of the instrument when the instrument is operating, and the instrument sound information includes information about the sound emitted by the instrument; The control module determines score information based on the instrument action information and / or the instrument sound information, and the score information is used to indicate the score section corresponding to the instrument action information or the instrument sound information; The mediumless holographic module adjusts a pre-generated mediumless holographic score image according to the score information; wherein, the mediumless holographic score image is floating imaging; determining score information based on the instrument action information includes: normalizing the instrument action information, where the instrument action information includes key pressing speed, strength, duration, action frequency, action amplitude; constructing the normalized instrument action information into an instrument action matrix, calculating the correlation value of the instrument action matrix according to the Pearson coefficient, if the correlation value is within a preset range, then preliminarily determine that the relevant action is reasonable; according to the instrument action matrix, matching the first matrix corresponding to a preset score with the highest similarity value to the instrument action matrix, and taking the preset score corresponding to the first matrix as the optional first score information, if the change in the actually played score is the same as the first score information, then taking the first score information as the final score information Determining score information based on the instrument sound information includes: normalizing the instrument sound information; constructing the normalized instrument sound information into an instrument sound matrix, calculating the correlation value of the instrument sound matrix according to the Pearson coefficient, if the correlation value is within a preset range, then preliminarily determine that the relevant sound is reasonable; according to the instrument sound matrix, matching the second matrix corresponding to a preset score with the highest similarity value to the instrument sound matrix, and taking the preset score corresponding to the second matrix as the optional second score information; if the change in the actually played score is the same as the second score information, then taking the second score information as the final score information.

6. The method according to claim 5, wherein Further includes: Collect external action information through a second sensor, where the external action information includes external action information of an external action operating on the instrument; The control module determines score information based on the external action information, the instrument action information, and / or the instrument sound information, and adjusts the mediumless holographic score image according to the score information.

7. The method according to claim 5, wherein The determining score information based on the instrument action information includes: Determining syllable information corresponding to the instrument action according to the instrument action information; Matching the syllable information with a preset score to determine the score information.

8. The method according to claim 5, characterized in that The determining score information based on the instrument sound information includes: Determining syllable information corresponding to the instrument sound according to the instrument sound information; Matching the syllable information with a preset score to determine the score information.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the method described in any one of claims 5 to 8 when running.

10. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method described in any one of claims 5 to 8.

Images