Music interaction system and method based on virtual reality

Abstract

The invention discloses a music interaction system and method based on virtual reality, and belongs to the technical field of virtual reality. The system comprises a single-tone interaction module used for simulating musical notes through single-tone interaction bodies corresponding to the musical notes in music, and the single-tone interaction bodies can conduct visual interaction guidance on a user; obtaining an interaction result according to the interaction time and mode of the user and the single-tone interaction body; the level visualization module is used for carrying out visualization output on the single-tone interaction body; when a certain note in the music is played, the corresponding single-tone interaction body is displayed according to a preset display time point, a pose and an expected interaction mode; and the numerical value management module is used for feeding back an interaction result between the user and the single-tone interaction body. According to the invention, the virtual reality technology is utilized to construct the single-tone interaction body to simulate the notes in the classical music, so that the user can practice, teach or simulate playing through interaction with the single-tone interaction body, and the teaching is more interesting.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual reality, and in particular relates to a music interaction system and method based on virtual reality. Background Art

[0002] At present, traditional music represented by classical music is gradually fading from the mainstream music stage. This type of music is known for its serious style. Its "dull" and "difficult to understand" characteristics have been deeply rooted in people's minds and it is difficult to meet the changing aesthetic needs of modern young people. On the other hand, learning classical instruments requires long-term practice and investment. The process is relatively boring and lacks immediate sense of achievement and entertainment. Existing music teaching software still lacks user feedback when providing music teaching services. At the same time, the software theme is too serious and lacks fun. Users usually cannot understand their performance level at the first time.

[0003] Virtual Reality (VR) is a computer-generated simulated environment that allows users to immerse themselves in and interact with it. In virtual reality, users can experience a three-dimensional virtual world by wearing special equipment such as virtual reality helmets, gloves or other sensors.

[0004] How to rekindle people’s interest in traditional music through innovative technological means such as VR technology is a technical problem that needs to be solved urgently.

[0005] Prior art Chinese patent application CN202010356201.0 discloses a music teaching and training system based on virtual reality, including a musical instrument audio management module, an audio waveform simulation module, a musical instrument type judgment module and a training instrument error correction module; the musical instrument audio management module is used to transmit the audio signal of the playing instrument; the audio waveform simulation module is used to simulate the audio waveform data of the transmitted audio signal; the musical instrument type judgment module is used to automatically determine the type of the playing instrument according to the simulated audio waveform.

[0006] In the above-mentioned prior art, the VR technology is mainly used to correct the user's performance, thereby improving the correction management during music training and performance, rather than guiding through the VR technology to improve the fun of music teaching and stimulate the interest of learners. Summary of the invention

[0007] The purpose of the present invention is to provide a music interaction system based on virtual reality, which partially solves or alleviates the above-mentioned deficiencies in the prior art and improves the interest of music teaching through VR technology.

[0008] In order to solve the above-mentioned technical problems, the present invention specifically adopts the following technical solutions: a music interaction system based on virtual reality, comprising: A single-tone interaction module is used to simulate the notes through a single-tone interaction body corresponding to the notes in the music, and the single-tone interaction body can provide visual interaction guidance to the user; and obtain the interaction result according to the time and method of the user's interaction with the single-tone interaction body; The level visualization module is used to visualize the single-note interactive body. When a note in the music is played, the corresponding single-note interactive body is displayed according to the preset display time point, posture and expected interaction mode. The numerical management module is used to provide feedback on the interaction results between the user and the single-tone interaction object.

[0009] As an improvement, the music interaction system includes at least one of a first mode, a second mode and a third mode; In the first mode, the single-tone interactive body simulated by the single-tone interactive module is provided with an animation of a musical instrument playing action; when the user interacts with the single-tone interactive body in the expected interactive manner, the musical instrument playing action can be executed; In the second mode, when the user interacts with the single-tone interactive body in the expected interactive mode, the corresponding instrument playing action can be performed; the corresponding virtual audience is displayed according to the interaction result between the user and the single-tone interactive body; and the single-tone interactive bodies of different instruments are interspersed in the same music; In the third mode, the single-tone interactive bodies that appear simultaneously are single-tone interactive bodies of two different musical instruments. When the user interacts with the single-tone interactive bodies according to the expected interaction method, both hands respectively perform playing actions of the two different musical instruments.

[0010] As an improvement, it also includes a custom level file; the custom level file includes single-note interactive body information and runtime data; the single-note interactive body information includes style information, data information and display information, wherein the style information includes shape, texture and interactive effects; the data information includes score, instrument type, and magnification; the display information includes display time point, posture and expected interaction method; the runtime data includes level duration and the number of single-note interactive bodies.

[0011] As an improvement, the monophonic interaction module includes: Art material setting module, used to set the style of the monophonic interactive body according to the style information specified in the custom level file; The value recording module is used to set the value of the single-tone interactive body according to the value information specified in the custom level file; The art asset module is used to model and render the single-tone interactive body, and to create the single-tone interactive body entry animation and interactive special effects, thereby forming a single-tone interactive body material library; An interaction determination module is used to record the time and method of interaction between a user and a single-tone interactive body, and obtain an interaction result based on the time and method of interaction; when the user interacts with the single-tone interactive body according to the expected time and interaction method, the interaction result is a successful interaction; when the user does not interact with the single-tone interactive body according to the expected time and / or interaction method, the interaction result is a failed interaction.

[0012] As an improvement, the level visualization module includes: An external file reading module is used to read an audio file and a corresponding custom level file, and call a dynamic audio module to play the audio file; the audio file contains music; The custom level file parsing module is used to parse the custom level file and convert the custom level file into data recognizable by the system, so that the system outputs the corresponding single-tone interaction body according to the custom file.

[0013] As an improvement, the level visualization module further includes: The standard interface module is used to provide a program interface for custom level files and generate custom level files using external programs.

[0014] As an improvement, the numerical management module includes: Level metadata module, used to read the single-voice interactive body numerical information and runtime data in the custom level file; The score calculation module is used to calculate the user score based on the numerical information of the single-tone interaction body and the interaction result between the user and the single-tone interaction body; specifically, the formula is used: ; ; Calculate the score; where S(N) is the final score of the current note, s(N) is the preset score, b(N) is the basic score of the instrument of the current note, acc(N) is the accuracy, interaction is 1, and interaction failure is 0; M f (N) is the note score multiplier, which is determined by the level creator. Usually, the multiplier for accentuated notes is higher. m(C) is the combo bonus multiplier, where C is the number of notes currently hit continuously. m(C) defaults to 1, and each consecutive note hit adds 0.02, up to a maximum of 1.5. The performance progress module is used to calculate the current performance progress and display the user's score after the performance is completed.

[0015] As an improvement, the numerical management module further includes: The combo number calculation module is used to record the user's current combo number and pass the combo number to the score calculation module as a parameter for calculating the user's score; the combo number is the number of successful interactions between the user and the single-tone interactive body within a threshold time interval.

[0016] The present invention also provides a music teaching interactive method based on virtual reality, comprising: The notes are simulated by a single-tone interactive body corresponding to the notes in the music, and the single-tone interactive body can provide visual interactive guidance to the user; The single-note interactive body is visualized and output; when a note in the music is played, its corresponding single-note interactive body is displayed according to the preset display time point, position and expected interaction mode; Obtaining the interaction result according to the time and manner of the user's interaction with the monophonic interactive object; Provide feedback on the interaction results between the user and the single-tone interactive object.

[0017] In the first mode, the single-tone interactive body simulated by the single-tone interactive module is provided with an animation of a musical instrument playing action; when the user interacts with the single-tone interactive body in the expected interactive manner, the musical instrument playing action can be executed; In the second mode, when the user interacts with the single-tone interactive body in the expected interactive mode, the corresponding instrument playing action can be performed; the corresponding virtual audience is displayed according to the interaction result between the user and the single-tone interactive body; and the single-tone interactive bodies of different instruments are interspersed in the same music; In the third mode, the single-tone interactive bodies that appear simultaneously are single-tone interactive bodies of two different musical instruments. When the user interacts with the single-tone interactive bodies according to the expected interaction method, both hands respectively perform playing actions of the two different musical instruments.

[0018] The present invention is beneficial in that: The present invention utilizes virtual reality technology to construct a single-tone interactive body to simulate the notes in classical music, allowing users to practice, teach or simulate performance through interaction with the single-tone interactive body, thereby making the relatively boring classical music teaching more interesting, allowing users to increase their interest in classical music, and thus making classical music easier to inherit.

[0019] The invention comprises three modes: teaching, playing and rhythm.

[0020] In the teaching mode, the single-tone interactive body simulated by the single-tone interactive module is provided with an animation of the musical instrument playing action; when the user interacts with the single-tone interactive body in the expected interactive mode, the musical instrument playing action can be performed. This mode can not only realize the teaching of classical music performance, but also improve the fun of teaching, achieving the purpose of combining education with entertainment.

[0021] In the performance mode, when the user interacts with the monophonic interactive body in the expected interaction mode, the corresponding instrument playing action can be performed; the corresponding virtual audience is displayed according to the interaction results between the user and the monophonic interactive body; and the monophonic interactive bodies of different instruments are interspersed in the same music. This mode simulates the instrument playing action to the greatest extent while taking into account the fun. The number of virtual audiences is used as feedback on the performance level to simulate the scene of the live performance. When designing a custom level file, the simulated performance of multiple instruments is interspersed in the entire music, and the performance of various instruments is simulated based on the position and interaction direction of the monophonic interactive body.

[0022] In rhythm mode, the single-tone interactive bodies that appear simultaneously are single-tone interactive bodies of two different musical instruments. When the user interacts with the single-tone interactive bodies in the expected interactive mode, both hands perform the playing actions of two different musical instruments respectively. In this mode, the main purpose is entertainment, so the difficulty of the level is increased. Two single-tone interactive bodies will appear for the same note in the music, and the two single-tone interactive bodies represent different musical instruments. Users need to simulate two instruments with both hands respectively, which increases the fun. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without paying creative labor.

[0024] Figure 1 This is a system principle diagram of Example 1 of the present invention.

[0025] Figure 2 This is a schematic diagram of the system module in Example 1 of the present invention.

[0026] Figure 3 A schematic diagram of the teaching mode.

[0027] Figure 4 This is a schematic diagram of the performance mode.

[0028] Figure 5 A schematic diagram of the rhythm pattern.

[0029] Figure 6 This is a flow chart of Example 2 of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] Herein, suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings by themselves. Therefore, "module", "component" or "unit" can be used mixedly.

[0032] In this document, the terms "upper", "lower", "inner", "outer", "front", "back", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0033] In this document, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0034] Herein "and / or" includes any and all combinations of one or more of the associated listed items.

[0035] Herein, "plurality" means two or more than two, ie, it includes two, three, four, five, etc.

[0036] Example 1: Figure 1 As shown, the present invention provides a music interaction system based on virtual reality, comprising: The single-tone interaction module is used to simulate the notes through the single-tone interaction body corresponding to the notes in the music, and the single-tone interaction body can provide visual interaction guidance to the user; and obtain the interaction result according to the time and method of the user's interaction with the single-tone interaction body.

[0037] In this embodiment, the notes in the music are simulated as "single-tone interactive bodies" through VR technology. Therefore, it can be foreseen that a set of VR equipment is required as hardware to deploy the present invention.

[0038] VR devices usually consist of the following main parts: 1. Display.

[0039] VR headsets are integrated with high-resolution displays to provide an immersive visual experience.

[0040] Sometimes two separate display screens are used (one for each eye) to achieve stereoscopic images.

[0041] Optical system: Includes lenses and optical components used to project images from the display into the user's eyes to create a three-dimensional stereoscopic effect.

[0042] 2. Sensor.

[0043] Positioning sensors: used to track the movement of the user's head, including accelerometers, gyroscopes, and magnetometers.

[0044] External tracking systems (such as base stations or cameras): used to accurately locate the headset and controllers in space.

[0045] 3. Audio system.

[0046] Built-in headphones or speakers provide spatial sound for enhanced immersion, and may also include a microphone for voice input.

[0047] 4. Input device.

[0048] Controller: Used to interact with the virtual environment, usually with buttons, touch panels, and motion sensors.

[0049] Sometimes other input devices are also supported, such as motion capture gloves, steering wheels, etc.

[0050] 5. Computing platform.

[0051] VR devices require high-performance computing power to generate images and process user input, typically including processing units inside PCs, game consoles, or standalone VR headsets.

[0052] The user can see the single-tone interactive body simulated by the system through the display screen, and interact with the single-tone interactive body through the controller under the interactive guidance of the single-tone interactive body.

[0053] For example, a single-note interactive body can be constructed in the shape of an arrow, and the direction of the arrow is used as an interaction guide. The user uses a controller to swipe the arrow in the direction indicated by the arrow within a specified time to complete the interaction of the single-note interactive body. By setting it in the shape of an arrow, on the one hand, it can better simulate the action of playing a musical instrument, and on the other hand, it can visually guide the user to strike the note according to the simulated action.

[0054] Of course, the form and interaction mode of the monophonic interaction body are various, and the present invention does not limit them.

[0055] In addition, this module also needs to judge the interaction results between the user and the single-voice interaction object. If the user interacts with it according to the guidance of the single-voice interaction within the set time, it is considered a successful interaction. If the user does not interact within the set time or / and does not interact according to the guidance, it is considered a failed interaction.

[0056] The level visualization module is used to visualize the single-note interactive body; when a note in the music is played, its corresponding single-note interactive body is displayed according to the preset display time point, posture and expected interaction method.

[0057] The level visualization module is used to present the monophonic interactive body on the display screen of the VR device in a preset manner. Since the present invention is a music interaction system, it is necessary to cultivate the user's musical sense and train the user's playing skills. Therefore, it is foreseeable that the appearance of the monophonic interactive body on the screen is synchronized with the notes played in the music. When the music plays to a certain note, the corresponding monophonic interactive body appears on the screen.

[0058] In the monophonic interaction module, the monophonic interaction body is constructed, and in the level visualization module, it is presented in a preset way, such as the time of appearance, the location of appearance, the direction of appearance, and the expected way of interaction.

[0059] For example, if the fifth note in the music is played at the fifth second, the single-note interactive object corresponding to the note (using the arrow as an example) will also appear on the screen at the fifth second. Of course, its appearance can have some appearance animations such as from far to near. In addition, if the expected interaction method of the single-note interactive object is to be slashed from top to bottom, the direction of the arrow is set to downward (in order to save resources and increase the reuse rate of materials, for example, only one arrow with different directions needs to be built, and then it can be rotated to different directions).

[0060] Of course, it is also necessary to set the interaction logic behind the single-tone interactive body so that when the user uses the controller to make a swiping action from top to bottom during the appearance of the single-tone interactive body, it is recorded as a successful interaction.

[0061] The numerical management module is used to provide feedback on the interaction results between the user and the single-tone interaction object.

[0062] The purpose of the present invention is to improve the fun of classical music performance, so it is necessary to set positive feedback to motivate users, thereby improving the user's inner pleasure.

[0063] This module calculates the score based on the interaction results between the user and the single-tone interaction object recorded by the single-tone interaction module, and is used to feedback the user's interaction performance.

[0064] This embodiment also includes a control module for controlling the above three modules to work in coordination.

[0065] In addition, due to the diversity of music, each piece of music can be practiced as a separate level in the present invention. And for the fun of practice, the corresponding single-tone interactive body in each level can be set to different styles, values ​​and display modes. In addition, the entire level itself can also be set, such as the duration of the level, the number of single-tone interactive bodies replaced, and so on. In order to record these settings so that each module can work according to these settings, the present embodiment also includes a custom level file; the custom level file includes single-tone interactive body information and runtime data; the single-tone interactive body information includes style information, data information and display information, wherein the style information includes shape, texture and interactive special effects; the data information includes score, instrument type, and magnification; the display information includes display time point, posture and expected interaction mode; the runtime data includes level duration and the number of single-tone interactive bodies.

[0066] Each piece of music corresponds to a custom level file. This system sets and displays the single-tone interactive body according to the custom level file, thereby forming a level that matches the music.

[0067] In addition, the system includes three modes, namely, a first mode, a second mode and a third mode. In some embodiments, the three modes may be a teaching mode, a performance mode and a rhythm mode.

[0068] like Figure 3 As shown, in the first mode, that is, the teaching mode, the single-tone interactive body simulated by the single-tone interactive module is provided with an animation of a musical instrument playing action; when the user interacts with the single-tone interactive body in the expected interactive manner, the musical instrument playing action can be executed.

[0069] For example, in the case of drumming, each note is a drum beat, and each drum beat corresponds to a single-note interactive body. In teaching mode, the single-note interactive body not only guides the interaction, but also carries the purpose of teaching. Therefore, on each single-note interactive body, an animation of the drum beat in the real performance is also set up, and it appears together with the single-note interactive body.

[0070] The interaction mode between the user and the single-tone interactive body is also set to the playing action. Taking drumming as an example, the expected interaction mode of each single-tone interactive body is to strike from top to bottom, and the animation of the drumstick also appears on the screen in synchronization with the user's action. Specifically, when the user strikes from top to bottom with the handheld controller, the drumstick in the animation also makes a striking action from top to bottom, thereby interacting with the single-tone interactive body.

[0071] Through the above method, we can not only realize the teaching of classical music performance, but also improve the interest of teaching, and achieve the purpose of combining education with entertainment.

[0072] This mode focuses on teaching music performance. By presenting the corresponding single instrument's performance animation on the single-note interactive body, users can unconsciously learn the performance skills of the instrument. When formulating a custom level file, it is necessary to reduce the difficulty and let users simulate the performance actions as much as possible.

[0073] Of course, different musical instruments have different playing actions. Therefore, in order to allow users to understand as many musical instruments as possible and have a preliminary understanding of the playing methods of the musical instruments, such as the most basic playing actions, different actions can be further set for different musical instruments (i.e., as expected interaction methods). For example, taking playing the violin as an example, the expected interaction method for each single-tone interaction is to draw a line from left to right, or from right to left, and an animation of playing the violin also appears on the screen in synchronization with the user's actions. Specifically, when the user draws a line from left to right through a handheld controller (for example, moving / rotating a joystick on the handheld controller from left to right), the bow in the animation also makes a corresponding pulling action, thereby interacting with the single-tone interaction body; or, when the user draws a line from right to left through a handheld controller (for example, moving / rotating a joystick on the handheld controller from right to left), the bow in the animation also makes a corresponding pulling action, thereby interacting with the single-tone interaction body.

[0074] like Figure 4 As shown, in the second mode, that is, the performance mode, when the user interacts with the single-tone interactive body according to the expected interaction method, the corresponding instrument playing action can be executed; the corresponding virtual audience is displayed according to the interaction result between the user and the single-tone interactive body; and single-tone interactive bodies of different instruments are interspersed in the same music.

[0075] In this mode, the same piece of music may be interspersed with single-tone interactive bodies of different musical instruments, such as single-tone interactive bodies of drums and single-tone interactive bodies of violins, etc. When the user interacts with the single-tone interactive body of drums, the action of playing the drums is performed. When the user interacts with the single-tone interactive body of violins, the action of playing the violin is performed.

[0076] In order to enhance the atmosphere, virtual audiences are added to this mode. The number of virtual audiences is determined by the user interaction results. The higher the interaction success rate, the more virtual audiences there are, giving users the experience of live performance.

[0077] This mode simulates the performance of musical instruments to the greatest extent while taking into account the fun of the performance. The number of virtual audience members is used as feedback on the performance level to simulate the scene of a live performance. When designing a custom level file, the entire piece of music is interspersed with simulated performances of multiple instruments, and the performances of various instruments are simulated based on the position and interaction direction of the single-tone interactive body. At the same time, the density of the single-tone interactive body can also be arranged according to the trend of the music's emotions.

[0078] like Figure 5 As shown, in the third mode, that is, the rhythm mode, the single-tone interactive bodies that appear simultaneously are single-tone interactive bodies of two different musical instruments. When the user interacts with the single-tone interactive bodies in the expected interaction manner, both hands perform the playing actions of the two different musical instruments respectively.

[0079] In this mode, the main purpose is entertainment, so the difficulty of the level is increased. The same note in the music will have two single-tone interactive bodies, and the two single-tone interactive bodies represent different instruments. Users need to simulate two instruments with both hands, which increases the fun and can intuitively compare the performance of two different instruments for the same note, allowing users to better understand different classical instruments.

[0080] In addition, this mode also focuses more on scoring, and adds new ways to obtain points, such as combos.

[0081] like Figure 2 As shown, the above modules are further elaborated below.

[0082] The monophonic interaction module comprises: The art material setting module is used to set the style of the single-tone interactive body according to the style information specified in the custom level file.

[0083] The styles of a single-tone interactive body are varied, such as arrows, musical notes, and any other shapes, and can even be an animation. The role of this submodule is to set the style of the single-tone interactive body according to the style information specified in the custom level file. The style of a single-tone interactive body specifically includes its shape, external texture, and interactive special effects. For example, the shape of a single-tone interactive body is an arrow, the texture is blue, and its interactive special effect is an explosion. The so-called interactive special effect refers to the animation that appears when the user interacts with the single-tone interactive body in the expected interactive mode. Taking drumming as an example, when the simulated drumstick hits the single-tone interactive body from top to bottom, it explodes.

[0084] The value recording module is used to set the value of the single-tone interactive body according to the value information specified in the custom level file.

[0085] The single-note interactive body is the interactive subject in the level, and its interaction result is the basis for the final score. Therefore, each single-note interactive body has a corresponding score and multiplier (for example, the general multiplier is 1 times, and the more difficult or more critical notes are 2 times, etc.). In addition, the instrument types of the single-note interactive body are also included, such as drums, violins, pianos, etc.

[0086] The art asset module is used to model and render single-tone interactive objects, and to create single-tone interactive object entry animations and interactive special effects, thereby forming a single-tone interactive object material library.

[0087] After the art material setting module sets up the monophonic interactive body according to the custom level file, the submodule models, renders and produces animations according to the settings. The produced monophonic interactive body is saved as a material library for the level visualization module to call.

[0088] An interaction determination module is used to record the time and method of interaction between a user and a single-tone interactive body, and obtain an interaction result based on the time and method of interaction; when the user interacts with the single-tone interactive body according to the expected time and interaction method, the interaction result is a successful interaction; when the user does not interact with the single-tone interactive body according to the expected time and / or interaction method, the interaction result is a failed interaction.

[0089] The function of this submodule is to determine whether the user interacts with the monophonic interactive object at the expected time and in the expected way. If so, the interaction is considered successful; if not, the interaction is considered failed. The interaction results are saved and used by the value management module as the basis for calculating the score.

[0090] Of course, the results of the interaction can include other factors besides success and failure. For example, a score of 0 to 100 can be set, with complete failure being 0 points, and interaction completely in accordance with requirements being 100 points. Some defects in the interaction will result in a partial deduction of the score, etc.

[0091] The level visualization module includes: The external file reading module is used to read the audio file and the corresponding custom level file, and call the dynamic audio module to play the audio file; the audio file contains music.

[0092] It is understandable that the custom level file and the audio file are read by the system as external files to form a level. The audio file includes music that can be played, and the custom level file sets the single tone interaction body corresponding to each note in the music according to the music, and also sets the level itself.

[0093] In order to enable audio playback, this submodule also includes a lower-level dynamic audio module for playing audio. There is no specific restriction on the audio format. The dynamic audio module includes a music dynamic import module and a music playback module; the music dynamic import module is responsible for converting audio data into a format that can be parsed by the engine used by this software, and the music playback module is used to play the parsed audio.

[0094] The custom level file parsing module is used to parse the custom level file and convert the custom level file into data recognizable by the system, so that the system outputs the corresponding single-tone interaction body according to the custom file.

[0095] The custom level file needs to be parsed by this module before it can be recognized by this system. It can be understood as converting a high-level language into low-level instructions that can be read by the system. The system calls the single-tone interactive body material in the art asset module according to the instructions, and outputs it to the screen in a set way, and has the expected interactive logic waiting for the user to interact.

[0096] The standard interface module is used to provide a program interface for custom level files and generate custom level files using external programs.

[0097] The standard interface module, as the API (application programming interface) of this system, can realize the generation and editing of custom level files through external programs. Of course, this system can also have a built-in custom level editing module, which is also not limited in the present invention.

[0098] The numerical management module comprises: The level metadata module is used to read the single-voice interactive body numerical information and runtime data in the custom level file.

[0099] The numerical information of a single-note interaction includes the score, multiplier, and instrument type. The runtime data of a level includes the level duration and the number of single-note interactions. After the numerical information is read and combined with the interaction results, the score of each single-note interaction can be calculated, and then the scores of all single-note interactions can be added together to get the score of the entire level.

[0100] The score calculation module is used to calculate the user score based on the numerical information of the single-tone interaction body and the interaction result between the user and the single-tone interaction body.

[0101] In this embodiment, the formula is used: ; ; Where S(N) is the final score of the current note, s(N) is the preset score, b(N) is the basic score of the instrument of the current note, acc(N) is the accuracy, 1 is the interaction success, and 0 is the interaction failure; Mf (N) is the note score multiplier, which is determined by the level creator, and usually the accented multiplier is higher; m(C) is the combo bonus multiplier, where C is the number of notes currently hit in a row.

[0102] The combo number calculation module is used to record the user's current combo number and pass the combo number to the score calculation module as a parameter for calculating the user's score; the combo number is the number of successful interactions between the user and the single-tone interactive body within a threshold time interval.

[0103] In order to increase the fun and form positive feedback for the user's successful performance, thereby promoting the user's in-depth learning and understanding of the corresponding musical instrument, a "combo" strategy is also formulated in this embodiment. For example, in a fast-paced music, a "combo" will be triggered if a threshold number of single-tone interactive bodies are successfully interacted with continuously, thereby increasing the score. For example, if 5 single-tone interactive bodies are successfully interacted with continuously, the scores of the 5 single-tone interactive bodies will be doubled.

[0104] The performance progress module is used to calculate the current performance progress and display the user's score after the performance is completed.

[0105] This module is used to manage the progress of the entire level. After the level is completed, the score calculated by the score calculation module is displayed on the screen.

[0106] Example 2: Figure 6 As shown, the present invention also provides a music teaching interactive method based on virtual reality, comprising: S1 simulates the notes through a single-tone interactive body corresponding to the notes in the music, and the single-tone interactive body can provide visual interactive guidance to the user; S2 single-tone interactive body is visualized and output; when a note in the music is played, its corresponding single-tone interactive body is displayed according to the preset display time point, position and expected interaction mode; S3 obtains the interaction result according to the time and manner of the user's interaction with the monophonic interaction object; S4 provides feedback on the interaction result between the user and the single-tone interaction object.

[0107] In the first mode, the single-tone interactive body simulated by the single-tone interactive module is provided with an animation of a musical instrument playing action; when the user interacts with the single-tone interactive body in the expected interactive manner, the musical instrument playing action can be executed; In the second mode, when the user interacts with the single-tone interactive body in the expected interactive mode, the corresponding instrument playing action can be performed; the corresponding virtual audience is displayed according to the interaction result between the user and the single-tone interactive body; and the single-tone interactive bodies of different instruments are interspersed in the same music; In the third mode, the single-tone interactive bodies that appear simultaneously are single-tone interactive bodies of two different musical instruments. When the user interacts with the single-tone interactive bodies according to the expected interaction method, both hands respectively perform playing actions of the two different musical instruments.

[0108] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a computer terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0110] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A music interaction system based on virtual reality, characterized in that include: A single-tone interaction module is used to simulate the notes through a single-tone interaction body corresponding to the notes in the music, and the single-tone interaction body can provide visual interaction guidance to the user; and obtain the interaction result according to the time and method of the user's interaction with the single-tone interaction body; The level visualization module is used to visualize the single-note interactive body. When a note in the music is played, the corresponding single-note interactive body is displayed according to the preset display time point, posture and expected interaction mode. The numerical management module is used to provide feedback on the interaction results between the user and the single-tone interaction object.

2. A music interaction system based on virtual reality according to claim 1, characterized in that: The music interaction system includes at least one of a first mode, a second mode and a third mode; In the first mode, an animation of a musical instrument playing action is set on the single-tone interactive body simulated by the single-tone interactive module; When the user interacts with the single-tone interactive object in the expected interactive mode, the musical instrument playing action may be performed; In the second mode, when the user interacts with the single-tone interactive body in the expected interactive mode, the corresponding instrument playing action can be performed; the corresponding virtual audience is displayed according to the interaction result between the user and the single-tone interactive body; and the single-tone interactive bodies of different instruments are interspersed in the same music; In the third mode, the single-tone interactive bodies that appear simultaneously are single-tone interactive bodies of two different musical instruments. When the user interacts with the single-tone interactive bodies according to the expected interaction method, both hands respectively perform playing actions of the two different musical instruments.

3. A music interaction system based on virtual reality according to claim 1, characterized in that: It also includes a custom level file; the custom level file includes single-note interactive body information and runtime data; the single-note interactive body information includes style information, data information and display information, wherein the style information includes shape, texture and interactive effects; the data information includes score, instrument type, and magnification; the display information includes display time point, posture and expected interaction method; the runtime data includes level duration and the number of single-note interactive bodies.

4. A music interaction system based on virtual reality according to claim 3, characterized in that The monophonic interaction module comprises: Art material setting module, used to set the style of the monophonic interactive body according to the style information specified in the custom level file; The value recording module is used to set the value of the single-tone interactive body according to the value information specified in the custom level file; The art asset module is used to model and render the single-tone interactive body, and to create the single-tone interactive body entry animation and interactive special effects, thereby forming a single-tone interactive body material library; An interaction determination module is used to record the time and method of interaction between a user and a single-tone interactive body, and obtain an interaction result based on the time and method of interaction; when the user interacts with the single-tone interactive body according to the expected time and interaction method, the interaction result is a successful interaction; when the user does not interact with the single-tone interactive body according to the expected time and / or interaction method, the interaction result is a failed interaction.

5. The music interaction system based on virtual reality according to claim 3, characterized in that The level visualization module includes: An external file reading module is used to read an audio file and a corresponding custom level file, and call a dynamic audio module to play the audio file; the audio file contains music; The custom level file parsing module is used to parse the custom level file and convert the custom level file into data recognizable by the system, so that the system outputs the corresponding single-tone interaction body according to the custom file.

6. A music interaction system based on virtual reality according to claim 5, characterized in that The level visualization module further includes: The standard interface module is used to provide a program interface for custom level files and generate custom level files using external programs.

7. The music interaction system based on virtual reality according to claim 3, characterized in that The numerical management module comprises: Level metadata module, used to read the single-voice interactive body numerical information and runtime data in the custom level file; The score calculation module is used to calculate the user score based on the numerical information of the single-tone interaction body and the interaction result between the user and the single-tone interaction body; specifically, the formula is used: ； ； Calculate the score; where S(N) is the final score of the current note, s(N) is the preset score, b(N) is the basic score of the instrument of the current note, acc(N) is the accuracy, 1 for successful interaction, and 0 for failed interaction; M f (N) is the note score multiplier, which is determined by the level creator. Usually, the multiplier for accented notes is higher. m(C) is the combo bonus multiplier, where C is the number of notes currently hit in a row. The performance progress module is used to calculate the current performance progress and display the user's score after the performance is completed.

8. A music interaction system based on virtual reality according to claim 7, characterized in that The numerical management module also includes: The combo number calculation module is used to record the user's current combo number and pass the combo number to the score calculation module as a parameter for calculating the user's score; the combo number is the number of successful interactions between the user and the single-tone interactive body within a threshold time interval.

9. A music teaching interactive method based on virtual reality, characterized in that include: The notes are simulated by a single-tone interactive body corresponding to the notes in the music, and the single-tone interactive body can provide visual interactive guidance to the user; The single-note interactive body is visualized and output; when a note in the music is played, its corresponding single-note interactive body is displayed according to the preset display time point, position and expected interaction mode; Obtaining the interaction result according to the time and manner of the user's interaction with the monophonic interactive object; Provide feedback on the interaction results between the user and the single-tone interactive object.

10. The music teaching interactive method based on virtual reality according to claim 9 is characterized in that The music teaching interactive method includes at least one of a first mode, a second mode and a third mode; In the first mode, an animation of a musical instrument playing action is set on the single-tone interactive body simulated by the single-tone interactive module; When the user interacts with the single-tone interactive object in the expected interactive mode, the musical instrument playing action may be performed; In the second mode, when the user interacts with the single-tone interactive body in the expected interactive mode, the corresponding instrument playing action can be performed; the corresponding virtual audience is displayed according to the interaction result between the user and the single-tone interactive body; and the single-tone interactive bodies of different instruments are interspersed in the same music; In the third mode, the single-tone interactive bodies that appear simultaneously are single-tone interactive bodies of two different musical instruments. When the user interacts with the single-tone interactive bodies according to the expected interaction method, both hands respectively perform playing actions of the two different musical instruments.

Images