Medium-free holographic equipment for musical instrument and control method of medium-free holographic equipment

Abstract

The embodiment of the invention provides medium-free holographic equipment for a musical instrument and a control method of the medium-free holographic equipment, and relates to the technical field of a music score information medium-free display technology. The medium-free holographic equipment for the musical instrument comprises a medium-free holographic module used for generating and displaying a medium-free holographic music score image; the first sensor is used for collecting musical instrument action information and / or musical instrument sound information of musical instrument work; and the control module is in signal connection with the first sensor and the medium-free holographic module and is used for determining music score information according to the musical instrument action information and / or the musical instrument sound information and adjusting the medium-free holographic music score image according to the music score information. According to the invention, the problem of high music score operation complexity is solved, and the effects of improving the playing effect and reducing playing errors are achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of medium-free display of music score information, and in particular to a medium-free holographic device for a musical instrument and a control method thereof. Background Art

[0002] In the field of musical instrument performance, sheet music is an important basis for performers to perform. In traditional musical instrument performance, performers usually need to manually flip through paper sheet music during the performance. This method not only easily distracts the performer's attention and affects the continuity and accuracy of the performance, but may also cause great psychological pressure on the performer during complex movements or fast performances. In addition, for some large-scale performances or professional performances, special staff are usually required to assist in turning pages, which not only increases labor costs, but may also cause performance interruptions due to operational errors, affecting the performance effect.

[0003] With the development of science and technology, electronic music score display devices have gradually been introduced into musical instrument performance. These devices display music scores through display screens, and can realize automatic page turning and adjustment of music scores, which to a certain extent solves the shortcomings of traditional paper music scores. However, most existing electronic music score display devices rely on touch screen operation or external control devices, and performers still need to be distracted to operate during the performance. In addition, the devices usually need to maintain a certain physical connection with the instrument, which limits the performer's range of movement and performance flexibility.

[0004] There is currently no better solution to the above problems. Summary of the invention

[0005] The embodiment of the present invention provides a medium-free holographic device for a musical instrument and a control method thereof, so as to at least solve the problem of high complexity of music score operation in the related art.

[0006] According to one embodiment of the present invention, there is provided a medium-free holographic device for a musical instrument, comprising: A medium-free holographic module, used for generating and displaying medium-free holographic music score images; A first sensor is used to collect musical instrument motion information and / or musical instrument sound information of the musical instrument, wherein the musical instrument motion information includes the vibration action or the pressing and pulling action performed by the keys or strings of the musical instrument when the musical instrument is working, and the musical instrument sound information includes the information of the sound emitted by the musical instrument; A control module is signal-connected to the first sensor and the medium-free holographic module, and is used to determine the music score information according to the musical instrument movement information and / or the musical instrument sound information, and adjust the medium-free holographic music score image according to the music score information.

[0007] In an exemplary embodiment, it also includes: A second sensor, connected to the control module by signal, for collecting external action information of operating the musical instrument by external action; The control module determines music score information according to the external action information, the musical instrument action information and / or the musical instrument sound information, and adjusts the medium-free holographic music score image according to the music score information.

[0008] In an exemplary embodiment, the medium-free holographic module comprises: A display element, connected to the control module by signal, and used to generate a music score image according to the instruction of the control module; The optical waveguide plate is used for optically processing the music score image to generate a medium-free holographic music score image.

[0009] In an exemplary embodiment, the musical instrument is connected to the body of the musical instrument using a medium-free holographic device; or, The medium-free holographic device for the musical instrument is not connected to the body of the musical instrument.

[0010] According to another embodiment of the present invention, there is provided a method for controlling a medium-free holographic device for a musical instrument, comprising: Acquiring musical instrument action information and / or musical instrument sound information of the musical instrument through the first sensor, wherein the musical instrument action information includes the vibration action or the pressing and pulling action performed by the keys or strings of the musical instrument when the musical instrument is working, and the musical instrument sound information includes the information of the sound emitted by the musical instrument; The control module determines the music score information according to the musical instrument action information and / or the musical instrument sound information, wherein the music score information is used to indicate the music score chapter corresponding to the musical instrument action information or the musical instrument sound information; The medium-free holographic module adjusts the pre-generated medium-free holographic music score image according to the music score information.

[0011] In an exemplary embodiment, it also includes: Collecting external action information through a second sensor, wherein the external action information includes external action information of operating the musical instrument by an external action; The control module determines the music score information according to the external action information, the musical instrument action information and / or the musical instrument sound information, and adjusts the medium-free holographic music score image according to the music score information.

[0012] In an exemplary embodiment, determining the music score information according to the musical instrument action information includes: Determining syllable information corresponding to the musical instrument action according to the musical instrument action information; The syllable information is matched with a preset music score to determine the music score information.

[0013] In an exemplary embodiment, determining the music score information according to the musical instrument sound information comprises: Determining syllable information corresponding to the instrument sound according to the instrument sound information; The syllable information is matched with a preset music score to determine the music score information.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0015] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] Through the present invention, since the sound or movement of the musical instrument is collected and recognized, the music score recognition and page turning can be automatically performed without manual operation. Therefore, the problem of high complexity of music score operation can be solved, and the effect of improving the performance of the musical instrument and reducing the difficulty of the performance of the musical instrument can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a structural block diagram of a medium-free holographic device for a musical instrument according to an embodiment of the present invention; Figure 2 It is a schematic diagram of the principle of a specific embodiment of the present invention. Figure 1 ; Figure 3 It is a schematic diagram of the principle of a specific embodiment of the present invention. Figure 2 ; Figure 4 It is a schematic diagram of the principle of a specific embodiment of the present invention. Figure 3 ; Figure 5 The present invention is a flowchart of a method for controlling a medium-free holographic device for a musical instrument according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0019] In the following, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0020] In addition, in the present application, directional terms such as "up", "down", "left" and "right" may be defined including but not limited to the orientation relative to the schematic placement of the components in the drawings. It should be understood that these directional terms may be relative concepts, which are used for relative description and clarification, and may change accordingly according to the change in the orientation of the components in the drawings.

[0021] In this application, unless otherwise specified or limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. In addition, the term "coupling" can be a way of achieving electrical connection for signal transmission.

[0022] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of variation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0023] In this embodiment, a medium-free holographic device for musical instruments is also provided. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0024] Generally, instrumentalists need to flip through the music scores when performing. This method places high demands on the performers for some complex pieces of music or pieces that need to be processed quickly. If the music score display is operated remotely by staff, the labor cost will be increased and it is easy to cause operational errors. Therefore, a new way of displaying music scores is needed to adapt to such performance occasions.

[0025] Example 1 Figure 1 is a structural block diagram of a medium-free holographic device 1 for a musical instrument according to an embodiment of the present invention. Figure 1 As shown, the medium-free holographic device 1 for musical instruments includes: A medium-free holographic module 11, a first sensor 12 and a control module 13, wherein the first sensor 12 is used to collect musical instrument action information and / or musical instrument sound information of the musical instrument, wherein the musical instrument action information includes the actions performed by the keys or strings of the musical instrument 2 when the musical instrument 2 is working, such as the vibration of the keys or strings of the musical instrument 2 themselves or the actions of the keys or strings of the musical instrument being pressed or pulled, etc., and the musical instrument sound information includes the information of the sound emitted by the musical instrument 2; the control module 13 is signal-connected to the first sensor 12 and the medium-free holographic module 11, and the medium-free holographic module 11 is used to generate and display a medium-free holographic music score image 113 (such as Figure 2 and Figure 3 As shown in the figure, the medium-free holographic score image 113 can be a five-line score or other types of scores (for example, dynamic operation prompts for indicating actions or keys, etc.); the control module 13 determines the score information according to the instrument action information and / or the instrument sound information, and adjusts the medium-free holographic score image 113 according to the score information. For example, the first sensor 12 collects a sound or action, and the control module 13 then recognizes that the sound or action corresponds to a certain section of the score, and then instructs the medium-free holographic module 11 to turn the page of the score or change the section, so that the performer can play the section without turning the page himself; in particular, in the process of turning the page or changing the section, the current action or syllable can be prompted according to the sound or action to prompt the current syllable position to ensure the correctness of the performance; of course, the prompting method can be a triangular indicator mark, a special color display, etc., which is not limited here, as long as the prompting can be performed, it falls within the scope of this invention.

[0026] Among them, the musical instrument 2 can be a keyboard instrument with certain keys such as a piano and an electronic keyboard, or a wind instrument such as a saxophone, a suona, and a wind instrument, a string instrument such as an erhu, a violin, and a harp, a percussion instrument such as a dulcimer and a drum, etc., that is, body-sounding instruments, membrane-sounding instruments, wind-sounding instruments, string-sounding instruments, and electric-sounding instruments are all within the scope of the musical instruments referred to in this application; the first sensor 12 can be a combination of an array microphone and a sound sensor for sound collection, or a high-pixel camera or a light sensor for motion collection, or other devices capable of sound or motion collection, such as a combination of a radar, a camera, and a sound sensor, a combination of an infrared sensor, a camera, and a sound sensor, a combination of a radar, a camera, an infrared sensor, and a sound sensor, etc., which are not limited here; the control module 13 can be a CPU / GPU combination for image and sound processing; the actions collected here include the actions of the musical instrument itself, such as the changes in the vibration of the keys / strings themselves when the keys are pressed, the strings are plucked or pulled, etc.

[0027] Of course, in order to recognize actions or sounds, it is necessary to use a trained neural network model or other deep learning model, which includes first denoising the collected sound information and / or action information to remove background noise and retain clear instrument sounds or instrument actions; then normalizing the data to standardize the data to the same range for better processing by the model; then extracting useful features from the original data, for example, for sound data, spectral features (such as MFCC, Mel-frequency cepstral coefficients) can be extracted, and for action data, the motion trajectory of key points can be extracted; then these data are input into the corresponding model, for example, a convolutional neural network (CNN) model suitable for processing image and video data, which can be used to recognize instrument actions, or a recurrent neural network (RNN) model suitable for processing time series data, which can recognize sound signals. Of course, related models of the Transformer architecture can also be used, which are not limited here; then the model automatically extracts features and makes predictions, Output the corresponding score information or syllable information, and match the recognized syllable information or score information with the preset score. The preset score can be stored in the database. Then, the recognized syllable sequence is compared with the syllable sequence in the score through a search algorithm (such as a dynamic time warping (DTW) algorithm) to find the best matching score chapter. Then, based on the matched score information, generate an instruction to adjust the medium-free holographic score image. For example, if it is recognized that the current music section needs to be turned, a page turning instruction is generated; if the current syllable position needs to be prompted, a syllable prompt instruction is generated; then the adjustment instruction is transmitted to the medium-free holographic module to update the content of the medium-free holographic score image by controlling the optical waveguide plate and the display component to keep it consistent with the performer's performance progress; wherein, in the model optimization process, if the match is successful, the position of the score currently being played is determined; if the match fails, a prompt message is issued to remind the performer or the system to perform further processing, such as further optimizing and updating the model according to the output result and the matching result.

[0028] like Figure 2 As shown, the medium-free holographic module 11 includes a display element 111 and an optical waveguide plate 112, wherein the display element 111 is signal-connected to the control module 13, and is used to generate a medium-free holographic music score image 113 according to the instructions of the control module; the display element 111 can be a display device or equipment such as a display screen or a display that can generate a light source image; the optical waveguide plate 112 is used to perform optical processing on the music score image to generate a medium-free holographic music score image 113; the optical waveguide plate 112 can be an optical element or device described in any Chinese patent with application numbers 202210060077.2, 202221492951.1, and 201920104395.8, or it can be other optical elements or devices that can form floating images, which are not limited here.

[0029] It should be noted that, due to the different types of musical instruments 2, the medium-free holographic device 1 for musical instruments can be connected to the body of the musical instrument 2, or it can be not connected to the body of the musical instrument 2. The connection here can be a direct connection such as a fixed connection or a detachable connection, or it can be a signal connection through a cable or a communication device. Among them, the fixed connection includes welding, gluing, and a connection method that is not easy to disassemble after connection through a specific installation structure. The detachable connection includes a connection method that can be disassembled and assembled by bolts, Velcro, etc.; the signal connection can be to install the first sensor 12 on the body of the musical instrument 2, and other modules are installed in other positions, and connected through a signal protocol or a cable. The transmission of the line signal is not limited here; it is easy to understand that the non-connection here means that the medium-free holographic device 1 for musical instruments has no direct contact with the body of the musical instrument 2, but directly collects the sound or action information of the musical instrument 2, and the scheme directly identifies it after screening the sound or action information; the body of the musical instrument 2 refers to the part that can be used to play the musical instrument, such as the body of a guitar, piano, violin, the body of a wind instrument, etc.; among them, if it is an instrument with a certain installation space such as a piano, the medium-free holographic device 1 for musical instruments can be installed in the body of the instrument. If it is an instrument without installation space such as a guitar, the medium-free holographic device 1 for musical instruments is not connected to the body of the musical instrument 2, as shown in the specific example. Figure 3-Figure 4 As shown, Figure 3 The structure (A) is that the medium-free holographic device 1 for a musical instrument is placed outside the body of the musical instrument 2 and connected to the body of the musical instrument 2; Figure 3 The structure in the middle (B) is that the medium-free holographic device 1 for a musical instrument is placed outside the body of the musical instrument and is not connected to the body of the musical instrument 2; Figure 4 The structure is that the medium-free holographic device 1 for musical instrument is placed outside the body of the musical instrument 2, but can be built-in or pushed and pulled by means of hinges; Figure 4 The structure in the figure is that the medium-free holographic device 1 for a musical instrument is built into the body of a musical instrument 2 without being hinged or pushed or pulled.

[0030] In particular, due to different installation locations and usage requirements, the image display and usage of medium-free holographic devices for musical instruments are also different. Take the piano as an example: like Figure 4 As shown in the structure (B), the medium-free holographic device 1 for a musical instrument can be fixed inside the piano body, and the medium-free music score image generated here can be located above the piano body or above the keys (i.e., in front of the piano body); or it can be installed on the upper surface of the piano body, and the medium-free music score image generated here can be located above the piano body or above the keys (i.e., in front of the piano body); or the medium-free holographic device 1 for a musical instrument can be built inside the piano body, and the medium-free holographic device 1 for a musical instrument can be hinged to the piano body (such as Figure 4As shown in the structure (A), the dotted part is an articulated, rotating, push-pull state), whereby the performer can push, pull or flip the medium-free holographic device for musical instruments according to their own needs, thereby changing the display position or display angle of the medium-free music score image, and the specific articulation method or installation method is not limited here, as long as the above-mentioned function can be achieved; of course, in addition to the articulation, other methods that can change the position of the built-in medium-free holographic device 1 for musical instruments and thus change the display position of the medium-free holographic music score image 113 all fall within the protection scope of the present application, such as methods of changing the position of the medium-free holographic device 1 for musical instruments by means of special sliding structures, fixed electronic push rods or cylinders and other devices or structures are all within the aforementioned scope and are not limited here.

[0031] Example 2 The difference from Example 1 is that, in order to further accurately identify the situation of the movement, the action of the performer can also be collected, and the progress of the movement can be identified according to the action of the performer; specifically, the medium-free holographic device for musical instruments also includes: A second sensor 14 is connected to the control module 13 by signal, and is used to collect external action information of operating the musical instrument by external action; The control module 13 determines the music score information according to the external action information, the musical instrument action information and / or the musical instrument sound information, and adjusts the medium-free holographic music score image 113 according to the music score information.

[0032] Generally, in a fixed movement, the performer's movements for the instrument are fixed, whether it is striking, blowing, plucking, pressing, or pulling, they all need to remain in a fixed position to achieve accurate performance of the movement. Therefore, at this time, by capturing and identifying the performer's movement changes and movement distribution over a period of time, the current movement situation can be determined, so as to judge whether it is necessary to turn the page or change the movement.

[0033] It is easy to understand that the second sensor 14 can also be a sensor that can capture the performer's movements, such as at least any one of an infrared sensor, a light sensor, a camera, a radar, etc., which is not limited here; and the installation position and installation method of the second sensor 14 are preferably based on being able to capture the performer's body movements, such as movements of fingers, elbows, arms and other limbs, and for more accurate recognition, multiple sensors need to be set at multiple locations; the external action information is the performer's body movements performed on the instrument, such as hitting, blowing, plucking, pressing, pulling and other actions.

[0034] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0035] Example 3 In this embodiment, a method for controlling a medium-free holographic device for a musical instrument is provided. Figure 5 is a flow chart of a method for controlling a medium-free holographic device for a musical instrument according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps: Step S51, acquiring instrument action information and / or instrument sound information of the instrument through the first sensor 12, wherein the instrument action information includes the vibration action or the pressing and pulling action performed by the keys or strings of the instrument when the instrument is working, and the instrument sound information includes the information of the sound emitted by the instrument; Step S52, the control module 13 determines music score information according to the musical instrument action information and / or the musical instrument sound information, wherein the music score information is used to indicate the music score chapter corresponding to the musical instrument action information or the musical instrument sound information; In step S53 , the medium-free holographic module 11 adjusts the pre-generated medium-free holographic music score image according to the music score information.

[0036] The execution subject of the above steps may be a base station, a terminal, etc., but is not limited thereto.

[0037] In an optional embodiment, the method further comprises: Step S54, collecting external action information through the second sensor 14, wherein the external action information includes external action information of operating the musical instrument by an external action; In step S55 , the control module 13 determines music score information according to the external action information, the instrument action information and / or the instrument sound information, and adjusts the medium-free holographic music score image according to the music score information.

[0038] Example 4 Determining the music score information according to the musical instrument action information includes: Step S521, determining the syllable information corresponding to the musical instrument action according to the musical instrument action information; Step S522: Match the syllable information with a preset music score to determine the music score information.

[0039] In this embodiment, the purpose of collecting the movements of the musical instrument is to determine the progress of the movement based on the movement of the musical instrument itself, thereby effectively and conveniently providing the performer with a progress reminder of the movement or automatically turning pages.

[0040] Specifically, the collected musical instrument action information includes the action of the musical instrument keys or strings, such as the speed, strength, and duration of the keys being pressed, or the amplitude and frequency of the strings being plucked; the collected action information is then parsed to extract characteristic parameters related to the syllables. For example, for keyboard instruments, the notes played can be inferred based on the order, speed, and duration of the keys being pressed; for string instruments, the pitch and intensity can be determined based on the frequency and amplitude of the strings being plucked; for guitar playing, the corresponding notes can be determined by detecting the plucking action and the string pressing position; then, based on the characteristic parameters obtained from the analysis, the syllable information corresponding to the musical instrument action is determined, for example, by analyzing the order and time intervals in which the keys are pressed, the sequence of notes played can be determined; by analyzing the frequency of the strings being plucked, the pitch can be determined, and then the syllable can be determined; the syllable information can include parameters such as the pitch, duration, and intensity of the notes, and these parameters together constitute the syllable information corresponding to the musical instrument action.

[0041] The preset score can be a standard score stored in a database, or a user-defined score; the preset score information is stored in the database of the system; the extracted syllable information is matched with the preset score. The dynamic time warping (DTW) algorithm, the longest common subsequence (LCS) algorithm or other similarity matching algorithms can be used to compare the similarity between the syllable information and the score. During the matching process, the system will compare the pitch, duration, intensity and other parameters of the syllable information with the notes in the score one by one to find the most matching part; the determined score information includes the pitch, duration, intensity and other parameters of the notes, as well as the arrangement order and movement structure of the notes. The score information can be stored in a digital format, such as using the music XML format or MIDI format, to facilitate the system to read and match; by matching the syllable information with the preset score, the score information of the current performance can be determined. The matching process can include comparisons of the duration, pitch, rhythm and other aspects of the notes to ensure that the generated score information is accurate.

[0042] Example 5 The difference from Example 4 is that, in order to better ensure the accuracy of data matching, the collected instrument movement information can be further analyzed, and the progress of the movement can be judged by analyzing whether the instrument movement information is normal.

[0043] Specifically, the collected instrument action information (such as key pressing speed, strength, duration, action frequency, action amplitude, etc.) can be normalized, and then the normalized instrument action information can be constructed into an instrument action matrix. Then, the correlation value of the instrument action matrix is ​​calculated based on the Pearson coefficient. If the correlation value is within a preset range, it is preliminarily determined that the relevant action is reasonable. Then, based on the instrument action matrix, the matrix corresponding to the preset music score with the highest similarity value to the instrument action matrix is ​​matched, and the preset music score corresponding to the matrix is ​​used as optional music score information. If the subsequent actual performance score changes are the same as the optional music score information, the optional music score information is used as the final music score information, and so on.

[0044] Example 6 In an optional embodiment, determining the music score information according to the musical instrument sound information includes: Step S523, determining the syllable information corresponding to the instrument sound according to the instrument sound information; Step S524: Match the syllable information with a preset music score to determine the music score information.

[0045] In this embodiment, the purpose of collecting the movements of musical instruments is to determine the progress of the movement based on the sound of the musical instruments, thereby effectively and conveniently providing the performer with a progress reminder of the movement or automatically turning pages.

[0046] Among them, the fundamental frequency of the sound signal can be extracted through autocorrelation algorithm, cepstrum analysis and other methods to determine the pitch of the note; the loudness of the note can be determined by analyzing the amplitude of the sound signal; the duration of the note can be determined by short-time energy analysis and other methods; the sound characteristics of different musical instruments can be distinguished by analyzing the spectral characteristics of the sound signal; then, based on the extracted feature vector, the sound signal is classified or regressed using a pre-trained sound recognition model (such as a deep learning model) to determine the syllable information corresponding to the instrument sound; for example, for the sound signal of a piano performance, the model can identify specific notes (such as C4, D4, etc.) and their duration based on the extracted pitch, intensity and duration features, where the syllable information can be represented as a series of notes and their corresponding duration and intensity, for example: [C4, 1 / 4], [E4, 1 / 4], [G4, 1 / 2], indicating that the C4 note (duration 1 / 4 beat), E4 note (duration 1 / 4 beat) and G4 note (duration 1 / 2 beat) were played. Then, according to the matching result, the extracted syllable information successfully matches the note sequence of the 1st measure, beat 1-3 in the score. At this time, there is no need to turn the page, and so on.

[0047] Example 7 The difference from Example 6 is that, in order to better ensure the accuracy of data matching, the collected instrument sound information can be further analyzed, and the progress of the movement can be judged by analyzing whether the instrument sound information is normal.

[0048] Specifically, the collected instrument action information (such as fundamental frequency, loudness, amplitude, duration, spectral characteristics, etc.) can be normalized, and then the normalized instrument sound information can be constructed into an instrument sound matrix. Then, the correlation value of the instrument sound matrix can be calculated based on the Pearson coefficient. If the correlation value is within a preset range, it is preliminarily determined that the relevant sound is reasonable. Then, based on the instrument sound matrix, the matrix corresponding to the preset music score with the highest similarity value to the instrument sound matrix is ​​matched, and the preset music score corresponding to the matrix is ​​used as optional music score information. Then, if the subsequent actual performance score changes are the same as the optional music score information, the optional music score information is used as the final music score information, and so on.

[0049] Example 8 The difference from Example 5 and Example 7 is that after obtaining the optional music score information according to the instrument action matrix and the instrument sound matrix, the similarity of the two music scores is calculated. When the similarity meets the requirements, the optional music score information can be directly used as the final music score information without comparing it with the actual performance situation; the optional music score information can also be compared with the actual performance situation to ensure the accuracy of the performance.

[0050] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0051] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0052] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0053] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0054] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0055] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0056] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0057] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0058] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0059] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0060] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A medium-free holographic device for musical instruments, characterized in that: include: A medium-free holographic module, used for generating and displaying medium-free holographic music score images; A first sensor is used to collect musical instrument motion information and / or musical instrument sound information of the musical instrument, wherein the musical instrument motion information includes the vibration action or the pressing and pulling action performed by the keys or strings of the musical instrument when the musical instrument is working, and the musical instrument sound information includes the information of the sound emitted by the musical instrument; A control module is signal-connected to the first sensor and the medium-free holographic module, and is used to determine the music score information according to the musical instrument movement information and / or the musical instrument sound information, and adjust the medium-free holographic music score image according to the music score information.

2. The medium-free holographic device for musical instruments according to claim 1, characterized in that: Also includes: A second sensor, connected to the control module by signal, for collecting external action information of operating the musical instrument by external action; The control module determines music score information according to the external action information, the musical instrument action information and / or the musical instrument sound information, and adjusts the medium-free holographic music score image according to the music score information.

3. The medium-free holographic device for musical instruments according to claim 1, characterized in that: The medium-free holographic module comprises: A display element, connected to the control module by signal, and used to generate a music score image according to the instruction of the control module; The optical waveguide plate is used for optically processing the music score image to generate a medium-free holographic music score image.

4. The medium-free holographic device for musical instruments according to claim 1, characterized in that: The musical instrument is connected to the body of the musical instrument by a medium-free holographic device; or, The medium-free holographic device for the musical instrument is not connected to the body of the musical instrument.

5. A method for controlling a medium-free holographic device for a musical instrument, characterized in that: include: Acquiring musical instrument action information and / or musical instrument sound information of the musical instrument through the first sensor, wherein the musical instrument action information includes the vibration action or the pressing and pulling action performed by the keys or strings of the musical instrument when the musical instrument is working, and the musical instrument sound information includes the information of the sound emitted by the musical instrument; The control module determines the music score information according to the musical instrument action information and / or the musical instrument sound information, wherein the music score information is used to indicate the music score chapter corresponding to the musical instrument action information or the musical instrument sound information; The medium-free holographic module adjusts the pre-generated medium-free holographic music score image according to the music score information.

6. The method according to claim 5, characterized in that Also includes: Collecting external action information through a second sensor, wherein the external action information includes external action information of operating the musical instrument by an external action; The control module determines the music score information according to the external action information, the musical instrument action information and / or the musical instrument sound information, and adjusts the medium-free holographic music score image according to the music score information.

7. The method according to claim 5, characterized in that Determining the music score information according to the musical instrument action information includes: Determining syllable information corresponding to the musical instrument action according to the musical instrument action information; The syllable information is matched with a preset music score to determine the music score information.

8. The method according to claim 5, characterized in that Determining the music score information according to the musical instrument sound information comprises: Determining syllable information corresponding to the instrument sound according to the instrument sound information; The syllable information is matched with a preset music score to determine the music score information.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 5 to 8 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 5 to 8.

Images