A virtual reality-based music interaction system and method

Abstract

The application discloses a music interaction system and method based on virtual reality, and belongs to the technical field of virtual reality. The system comprises a single tone interaction module, which is used for simulating notes in a piece of music through single tone interaction bodies corresponding to the notes, the single tone interaction bodies can visually guide the interaction of users, and the interaction results are obtained according to the time and mode of the interaction between the users and the single tone interaction bodies; a level visualization module is used for visually outputting the single tone interaction bodies; when a certain note in the piece of music is played, the corresponding single tone interaction body is displayed according to preset display time points, poses and expected interaction modes; and a numerical management module is used for feeding back the interaction results between the users and the single tone interaction bodies. The single tone interaction bodies are simulated through virtual reality technology to simulate the notes in a piece of classical music, the users practice, teach or simulate the performance through the interaction with the single tone interaction bodies, and the teaching is more interesting.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of virtual reality, and particularly relates to a music interaction system and method based on virtual reality. BACKGROUND

[0002] Currently, traditional style music represented by classical music is gradually fading out of the stage of mainstream music. This kind of music is known for its serious style, and its characteristics of "stiffness" and "difficulty to understand" have been deeply rooted in people's hearts, and it is difficult to meet the changing aesthetic needs of modern young people. On the other hand, learning a classical musical instrument requires long-term practice and investment, and the process is relatively dull, lacking instant sense of achievement and entertainment. The existing music teaching software still lacks user feedback when providing music teaching services, and the software theme is too serious and lacks interest, so users usually cannot understand their own playing level at the first time.

[0003] Virtual reality (VR) technology is a technology that simulates an environment generated by a computer, which can immerse users in it and interact with it. In virtual reality, users can experience a three-dimensional virtual world by wearing special devices such as virtual reality headsets, gloves or other sensors.

[0004] How to rekindle people's interest in traditional music through innovative technological means such as VR technology is a technical problem that needs to be solved urgently.

[0005] The prior art Chinese patent application CN202010356201.0 discloses a music teaching and training system based on virtual reality, which includes a musical instrument audio management module, an audio waveform graph simulation module, a musical instrument type judgment module and a training musical instrument error correction module. The musical instrument audio management module is used for transmitting the audio signal of the playing musical instrument. The audio waveform graph simulation module is used for simulating the audio waveform graph data of the transmitted audio signal. The musical instrument type judgment module is used for automatically judging the type of the playing musical instrument according to the simulated audio waveform graph.

[0006] In the above-mentioned prior art, the user's performance is corrected through VR technology, thereby improving the correction management during music training performance. However, the VR technology is not used for guidance to improve the interest of music teaching so as to stimulate the interest of learners. SUMMARY

[0007] The present application aims to provide a music interaction system based on virtual reality, which partially solves or alleviates the above-mentioned deficiencies in the prior art, and improves the interest of music teaching through VR technology.

[0008] In order to solve the above-mentioned technical problems, the application specifically adopts the following technical scheme: a music interaction system based on virtual reality, comprising:

[0009] A single-tone interaction module is configured to simulate notes by single-tone interaction bodies corresponding to the notes in a piece of music, the single-tone interaction bodies can visually guide the user to interact, and the interaction results are obtained according to the time and manner of the user interacting with the single-tone interaction bodies;

[0010] A level visualization module is configured to visually output the single-tone interaction bodies, and when a note in the piece of music is played, the single-tone interaction body corresponding to the note is displayed according to a preset display time point, pose and expected interaction manner;

[0011] A numerical value management module is configured to feed back the interaction results between the user and the single-tone interaction bodies.

[0012] As an improvement, the music interaction system comprises at least one of a first mode, a second mode and a third mode;

[0013] In the first mode, the single-tone interaction bodies simulated by the single-tone interaction module are provided with animations of instrument playing actions, and when the user interacts with the single-tone interaction bodies according to the expected interaction manner, the instrument playing actions can be performed;

[0014] In the second mode, when the user interacts with the single-tone interaction bodies according to the expected interaction manner, the corresponding instrument playing actions can be performed, the corresponding virtual audience is displayed according to the interaction results of the user and the single-tone interaction bodies, and single-tone interaction bodies of different instruments are interspersed in the same piece of music;

[0015] In the third mode, the single-tone interaction bodies appearing at the same time are single-tone interaction bodies of two different instruments, and when the user interacts with the single-tone interaction bodies according to the expected interaction manner, the two hands perform playing actions of two different instruments respectively.

[0016] As an improvement, a custom level file is further included, the custom level file comprises single-tone interaction body information and runtime data, the single-tone interaction body information comprises style information, data information and display information, the style information comprises shape, texture and interaction special effects, the data information comprises score, instrument type and multiplier, and the display information comprises display time point, pose and expected interaction manner, and the runtime data comprises level duration and number of single-tone interaction bodies.

[0017] As an improvement, the single-tone interaction module comprises:

[0018] An art material setting module is configured to set the style of the single-tone interaction bodies according to the style information in the custom level file;

[0019] A numerical record module is configured to set the numerical value of the single-tone interactive body according to the numerical information defined in the custom level file;

[0020] An art asset module is configured to model and render the single-tone interactive body, and make approach animation and interactive special effects of the single-tone interactive body, thereby forming a single-tone interactive body material library;

[0021] An interaction judgment module is configured to record the time and manner of the user interacting with the single-tone interactive body, and obtain an interaction result according to the time and manner of the interaction; in the case that the user interacts with the single-tone interactive body in the expected time and manner, the interaction result is a successful interaction; in the case that the user does not interact with the single-tone interactive body in the expected time and / or manner, the interaction result is a failed interaction.

[0022] As an improvement, the level visualization module comprises:

[0023] An external file reading module is configured to read an audio file and a corresponding custom level file, and call the dynamic audio module to play the audio file; the audio file contains a piece of music.

[0024] A custom level file analysis module is configured to analyze the custom level file, convert the custom level file into data recognizable by the system, and enable the system to output a corresponding single-tone interactive body according to the custom file.

[0025] As an improvement, the level visualization module further comprises:

[0026] A standard interface module is configured to provide a program interface for the custom level file, and generate the custom level file by using an external program.

[0027] As an improvement, the numerical management module comprises:

[0028] A level metadata module is configured to read numerical information and runtime data of the single-tone interactive body in the custom level file.

[0029] A score calculation module is configured to calculate a user score based on the numerical information of the single-tone interactive body and the interaction result between the user and the single-tone interactive body; specifically, a formula is used:

[0030] ;

[0031] ;

[0032] to calculate the score; wherein S(N) is the final score value of the current note, s(N) is a preset score value, b(N) is the basic score of the instrument of the current note, acc(N) is the accuracy, and the interaction is 1 for success and 0 for failure; M f(N) is a note score multiplier, determined by the level creator, usually the heavy note has a higher multiplier; m(C) is a combo bonus multiplier, C is the current number of consecutive notes hit, m(C) defaults to 1, increases by 0.02 for each consecutive note hit, and the maximum is 1.5;

[0033] The performance progress module is configured to calculate the current performance progress and display the user score after the performance ends.

[0034] As an improvement, the numerical management module further comprises:

[0035] The combo number calculation module is configured to record the current combo number of the user and pass the combo number to the score calculation module as a parameter for calculating the user score; the combo number is the number of times that the user successfully interacts with the single note interaction body within a threshold time interval.

[0036] The present application also provides a virtual reality-based music teaching interaction method, comprising:

[0037] The single note interaction body corresponding to the notes in the music piece is used to simulate the notes, and the single note interaction body can visually guide the user.

[0038] The single note interaction body is visually output; when a note in the music piece is played, the corresponding single note interaction body is displayed according to the preset display time point, pose, and expected interaction mode.

[0039] The interaction result is obtained according to the time and mode of the user interacting with the single note interaction body.

[0040] The interaction result between the user and the single note interaction body is fed back.

[0041] In the first mode, the single note interaction body simulated by the single note interaction module is provided with an animation of a musical instrument performance action; when the user interacts with the single note interaction body according to the expected interaction mode, the musical instrument performance action can be performed.

[0042] In the second mode, when the user interacts with the single note interaction body according to the expected interaction mode, the corresponding musical instrument performance action can be performed; the corresponding virtual audience is displayed according to the interaction result of the user and the single note interaction body; and single note interaction bodies of different musical instruments are interspersed in the same music piece.

[0043] In the third mode, the simultaneously appearing single note interaction bodies are single note interaction bodies of two different musical instruments; when the user interacts with the single note interaction bodies according to the expected interaction mode, the user performs the performance actions of the two different musical instruments with both hands.

[0044] The present application has the following advantages:

[0045] The present application uses virtual reality technology to construct a single-tone interactive body to simulate notes in a classical music piece, so that users can practice, teach or simulate performance through interaction with the single-tone interactive body, so that the relatively boring classical music teaching is more interesting, so that users improve their interest in classical music, so that classical music is more easily inherited.

[0046] The present application includes three modes of teaching, performance and rhythm.

[0047] In the teaching mode, the single-tone interactive module simulates a single-tone interactive body provided with an animation of a musical instrument performance action; when the user interacts with the single-tone interactive body in the expected interaction mode, the musical instrument performance action can be performed. This mode can not only realize the teaching of classical music performance, but also improve the interest of teaching, and realize the purpose of combining teaching with entertainment.

[0048] In the performance mode, when the user interacts with the single-tone interactive body in the expected interaction mode, the corresponding musical instrument performance action can be performed; the corresponding virtual audience is displayed according to the interaction result of the user and the single-tone interactive body; and different musical instrument single-tone interactive bodies are interspersed in the same music piece. This mode maximizes the simulation of musical instrument performance actions while taking into account the interest. The number of virtual audiences is used as feedback of the performance level to simulate the scene of a live performance. When designing a custom level file, the simulation performance of multiple musical instruments is interspersed in the entire music piece, and the performance of various musical instruments is simulated based on the position and interaction direction of the single-tone interactive body.

[0049] In the rhythm mode, the single-tone interactive bodies appearing at the same time are single-tone interactive bodies of two different musical instruments, and the user performs two different musical instrument performance actions with both hands when interacting with the single-tone interactive bodies in the expected interaction mode. In this mode, entertainment is the main purpose, so the difficulty of the level is increased. The same note in the music piece will appear two single-tone interactive bodies, and the two single-tone interactive bodies represent different musical instruments. The user needs to simulate two musical instruments with both hands, thereby improving the interest. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. In all the drawings, similar elements or parts are generally identified by similar reference signs. In the drawings, each element or part is not necessarily drawn according to the actual proportion. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without paying creative labor.

[0051] Figure 1 The system schematic diagram of the present application embodiment 1.

[0052] Figure 2 is a schematic diagram of the system module in Embodiment 1 of the present application.

[0053] Figure 3 is a schematic diagram of the teaching mode.

[0054] Figure 4 is a schematic diagram of the performance mode.

[0055] Figure 5 is a schematic diagram of the rhythm mode.

[0056] Figure 6 is a flowchart of Embodiment 2 of the present application. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0058] Herein, the suffix such as "module", "part" or "unit" used for indicating an element is only for the convenience of description of the present application, and has no specific meaning by itself. Therefore, "module", "part" or "unit" can be used mixedly.

[0059] Herein, the terms "upper", "lower", "inner", "outer", "front", "back", "one end", "the other end" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of description of the present application and simplification of the description, and do not indicate or imply that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0060] Herein, unless otherwise explicitly specified and limited, the terms "mount", "provided with", "connected" and the like should be understood broadly, for example, "connected" can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be directly connected, or indirectly connected through an intermediate medium, can be the communication inside two elements. For a person of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0061] Herein, "and / or" includes any and all combinations of one or more listed related items.

[0062] "Plural" means two or more, i.e., including two, three, four, five, etc.

[0063] Embodiment 1: As shown, the present application provides a virtual reality-based music interaction system, comprising: Figure 1

[0064] Single tone interaction module, for simulating notes in a piece of music through single tone interaction bodies corresponding to the notes, the single tone interaction bodies can provide visual interaction guidance to the user; and obtaining interaction results according to the time and manner of user interaction with the single tone interaction bodies.

[0065] In this embodiment, the notes in the music are simulated as "single tone interaction bodies" through VR technology. Therefore, it can be foreseen that a set of VR equipment is needed as hardware to deploy the present application.

[0066] VR equipment is usually composed of the following main parts:

[0067] 1. Display.

[0068] High-resolution display screens are integrated in VR headsets to provide immersive visual experiences.

[0069] Sometimes two separate display screens are used (one for each eye) to achieve stereoscopic images.

[0070] Optical system:

[0071] Including lenses and optical components, used to project images in the display into the user's eyes to create a three-dimensional stereoscopic effect.

[0072] 2. Sensors.

[0073] Positioning sensors: used to track the movement of the user's head, including accelerometers, gyroscopes, and magnetometers, etc.

[0074] External tracking systems (such as base stations or cameras): used to accurately locate the position of the headset and controller in space.

[0075] 3. Audio system.

[0076] Built-in earphones or speakers provide spatial sound effects to enhance immersion, and may also include microphones for voice input.

[0077] 4. Input devices.

[0078] Controller: used to interact with the virtual environment, usually with buttons, touch panels, and motion sensors.

[0079] Sometimes other input devices are also supported, such as motion capture gloves, steering wheels, etc.

[0080] ​5、Computing platform.

[0081] VR devices require high-performance computing capabilities to generate images and process user input, usually including processing units inside PCs, game consoles, or standalone VR headsets.

[0082] The user can see the single-note interactive body simulated by the system through the display screen and interact with the single-note interactive body under the interactive guidance of the single-note interactive body through the controller.

[0083] For example, the single-note interactive body can be constructed in the shape of an arrow, and the direction of the arrow serves as the interactive guidance. The user strikes the arrow within a specified time according to the direction indicated by the arrow through the controller, thereby completing the interaction with the single-note interactive body. By setting the shape of an arrow, on the one hand, the action of playing a musical instrument can be better simulated, and on the other hand, the user is visually guided to strike the note according to the simulated action.

[0084] Of course, the form of the single-note interactive body and the interaction method are various, and the present application does not limit them.

[0085] In addition, the module also needs to judge the interaction result of the user and the single-note interactive body. If the user interacts with the single-note interactive body according to the guidance of the single-note interactive body within the specified time, it is considered to be a successful interaction. If the user does not interact within the specified time or / and does not interact in the guided manner, it is considered to be an interaction failure.

[0086] The level visualization module is used to visually output the single-note interactive body. When a note in the music is played, the corresponding single-note interactive body is displayed according to the preset display time point, pose, and expected interaction method.

[0087] The level visualization module is used to present the single-note interactive body on the display screen of the VR device in a preset manner. Since the present application is a music interaction system that needs to cultivate the musical sense of the user and train the playing skills of the user, it can be foreseen that the appearance of the single-note interactive body on the screen is synchronized with the notes played in the music. When the music plays to a certain note, the corresponding single-note interactive body appears on the screen.

[0088] In the single-note interaction module, the single-note interactive body is constructed, and in the level visualization module, it is presented in a preset manner. For example, the time of appearance, the position of appearance, the direction of appearance, and the expected interaction method.

[0089] For example, the 5th note in a piece of music is played at the 5th second, then the monophonic interactive body (also take the arrow as an example) corresponding to the note also appears on the screen at the 5th second. Of course, it can have some appearance animation such as from far to near. In addition, if the expected interaction mode of the monophonic interactive body is to be split from top to bottom, then the direction of the arrow is set to be downward (in order to save resources, provide the reuse rate of materials, for example, different direction arrows only need to be built one, and then rotated to different directions).

[0090] Of course, the interaction logic behind the monophonic interactive body also needs to be set, so that when the user makes a split action from top to bottom using the controller during the appearance of the monophonic interactive body, it is recorded as a successful interaction.

[0091] The numerical management module is used to feed back the interaction result between the user and the monophonic interactive body.

[0092] The purpose of the present application is to improve the interest of classical music performance, so it is necessary to set positive feedback to encourage users and improve the user's inner pleasure.

[0093] The module calculates the score according to the interaction result of the user and the monophonic interactive body recorded by the monophonic interactive module, and is used to feed back the interaction result of the user.

[0094] The embodiment also includes a control module for controlling the cooperative work of the above three modules.

[0095] In addition, due to the diversity of music, each piece of music can be used as a separate level for practice in the present application. In order to improve the interest of practice, the corresponding monophonic interactive body in each level can be set to different styles, values and display modes. In addition, the entire level itself can also be set, such as the duration of the level, the number of monophonic interactive bodies covered, etc. In order to record these settings, so that each module can work according to these settings, the embodiment also includes a custom level file; the custom level file includes monophonic interactive body information and runtime data; the monophonic interactive body information includes style information, data information and display information, wherein the style information includes shape, texture and interactive special effect; the data information includes score, instrument type, and multiplier; the display information includes display time point, pose and expected interaction mode; the runtime data includes level duration and monophonic interactive body quantity.

[0096] Each piece of music corresponds to a custom level file. The system sets and displays the monophonic interactive body according to the custom level file, thereby forming a level cooperating with the music.

[0097] In addition, the system includes three modes, i.e., a first mode, a second mode, and a third mode. In some embodiments, the three modes can be a teaching mode, a playing mode, and a rhythm mode.

[0098] As shown in FIG. 1, in the first mode, i.e., the teaching mode, the single-tone interaction module simulates a single-tone interaction body on which an animation of a musical instrument playing action is set; when a user interacts with the single-tone interaction body in an expected interaction manner, the musical instrument playing action can be performed. Figure 3

[0099] For example, taking drum playing as an example, each note is a drum beat, and each drum beat corresponds to a single-tone interaction body. In the teaching mode, the single-tone interaction body not only guides the interaction, but also serves the purpose of teaching. Therefore, an animation of a striking action of the drum beat in real playing is also set on each single-tone interaction body and appears together with the single-tone interaction body.

[0100] For example, taking drum playing as an example, each note is a drum beat, and each drum beat corresponds to a single-tone interaction body. In the teaching mode, the single-tone interaction body not only guides the interaction, but also serves the purpose of teaching. Therefore, an animation of a striking action of the drum beat in real playing is also set on each single-tone interaction body and appears together with the single-tone interaction body.

[0101] By the above-mentioned manner, the teaching of classical music playing can be realized, the interest of teaching can be improved, and the purpose of combining teaching with fun can be achieved.

[0102] The mode mainly serves the purpose of music playing teaching, and through presenting an animation of a playing action of a corresponding single musical instrument on a single-tone interaction body, a user can unconsciously learn the playing skill of the musical instrument. When a custom level file is made, the difficulty needs to be reduced, and the user is allowed to simulate the playing action as much as possible.

[0103] ​Of course, different musical instruments have different playing actions, and therefore, in order to enable the user to learn more about the musical instruments and preliminarily understand the playing methods of the musical instruments, such as the most basic playing actions, different actions (i.e., as the expected interaction mode) can be further set for different musical instruments. For example, taking a violin as an example, the expected interaction mode of each monophonic interactive body is to draw a line from left to right or from right to left, and a violin animation appears in the screen and is synchronized with the user action. Specifically, when the user draws a line from left to right (e.g., moves / rotates a control lever on the handheld controller from left to right) through the handheld controller, the bow in the animation also performs a corresponding drawing action to interact with the monophonic interactive body; or when the user draws a line from right to left (e.g., moves / rotates a control lever on the handheld controller from right to left) through the handheld controller, the bow in the animation also performs a corresponding drawing action to interact with the monophonic interactive body.

[0104] As shown in FIG. 2, in the second mode, i.e., the playing mode, when the user interacts with the monophonic interactive body according to the expected interaction mode, the corresponding musical instrument playing action can be performed; the corresponding virtual audience is displayed according to the interaction result of the user and the monophonic interactive body; and the monophonic interactive bodies of different musical instruments are interspersed in the same musical piece. Figure 4

[0105] In this mode, the monophonic interactive bodies of different musical instruments, such as the monophonic interactive body of a drum and the monophonic interactive body of a violin, can be interspersed in the same musical piece. When the user interacts according to the monophonic interactive body of the drum, a drum playing action is performed. When the user interacts according to the monophonic interactive body of the violin, a violin playing action is performed.

[0106] In order to improve the atmosphere, virtual audiences are also added in this mode. The number of virtual audiences is determined according to the user interaction result. The higher the interaction success rate is, the more virtual audiences there are, so that the user has the experience of live performance.

[0107] This mode maximizes the simulation of musical instrument playing actions while taking into account the interest. The number of virtual audiences is used as feedback of the playing level to simulate the scene of live performance. When designing a custom level file, simulated performances of multiple musical instruments are interspersed in the entire musical piece, and the playing of various musical instruments is simulated based on the position and interaction direction of the monophonic interactive body. In addition, the density of the monophonic interactive bodies can be arranged according to the trend of the music emotion.

[0108] As shown in FIG. 3, in the third mode, i.e., the rhythm mode, the monophonic interactive bodies that appear at the same time are the monophonic interactive bodies of two different musical instruments, and when the user interacts with the monophonic interactive bodies according to the expected interaction mode, the two hands perform the playing actions of the two different musical instruments, respectively. Figure 5

[0109] ​​In this mode, the difficulty of the level is increased, and the entertainment is the main purpose, so the same note in the music will appear two monophonic interactive bodies, which represent different musical instruments. The user needs to simulate two musical instruments with both hands, which improves the interest and enables the user to intuitively compare the playing effects of two different musical instruments on the same note, so that the user can understand different classical musical instruments better.

[0110] In addition, this mode also pays more attention to scoring, and increases the score acquisition methods such as continuous hitting.

[0111] As shown in the following, the above modules are further described. Figure 2

[0112] The monophonic interactive module includes:

[0113] The art material setting module is used to set the style of the monophonic interactive body according to the style information in the custom level file.

[0114] The style of the monophonic interactive body is various, such as arrow shape, note shape, and any other shape, or even an animation. The function of this sub-module is to set the style of the monophonic interactive body according to the style information in the custom level file. The style of the monophonic interactive body specifically includes its shape, external texture, and interactive special effect. For example, the shape of a monophonic interactive body is an arrow, the texture is blue, and the interactive special effect is explosion. The interactive special effect refers to the animation that appears when the user interacts with the monophonic interactive body in the expected interactive manner. Taking drumming as an example, when the simulated drumstick hits the monophonic interactive body from top to bottom, it explodes.

[0115] The numerical value recording module is used to set the numerical value of the monophonic interactive body according to the numerical value information in the custom level file.

[0116] The monophonic interactive body is the interactive subject in the level, and its interactive result is the basis for the final score. Therefore, each monophonic interactive body has a corresponding score, a multiplier (for example, the general multiplier is 1 times, and the more difficult or more critical notes are 2 times, etc.), and the type of musical instrument such as drum, violin, piano, etc.

[0117] The art asset module is used to model and render the monophonic interactive body, and make the monophonic interactive body approach animation and interactive special effect, so as to form a monophonic interactive body material library.

[0118] After the art material setting module sets the monophonic interactive body according to the custom level file, this sub-module models and renders according to the settings and makes the animation. The made monophonic interactive body is saved as a material library for the level visualization module to call.

[0119] ​An interaction judging module is configured to record the time and manner of the user's interaction with the monophonic interactive body and obtain an interaction result according to the time and manner of the interaction; in the case that the user interacts with the monophonic interactive body in the expected time and manner, the interaction result is a successful interaction; in the case that the user does not interact with the monophonic interactive body in the expected time and / or manner, the interaction result is a failed interaction.

[0120] The function of the sub-module is to determine whether the user interacts with the monophonic interactive body in the expected time and manner, and if so, the interaction is considered successful; if not, the interaction is considered failed. The interaction result is saved for the numerical management module to call as the basis for calculating the score.

[0121] Of course, the result of the interaction can include other than success and failure, for example, setting a score of 0-100, a complete failure being 0 points, and a complete interaction according to the requirements being 100 points, and some other interaction defects deducting a part of the score.

[0122] The level visualization module includes:

[0123] An external file reading module is configured to read an audio file and a corresponding custom level file, and call a dynamic audio module to play the audio file; the audio file contains music.

[0124] It can be understood that the custom level file and the audio file are read by the system as external files to form a level. The audio file includes music that can be played, and the custom level file sets the monophonic interactive body corresponding to each note according to the music, and also sets the level itself.

[0125] In order to enable the audio to be played, the sub-module further includes a lower dynamic audio module for playing the audio. The audio format is not specifically limited. The dynamic audio module includes a music dynamic import module and a music playing module; the music dynamic import module is responsible for converting the audio data into a format that can be parsed by the engine used by the software, and the music playing module is used to play the parsed audio.

[0126] A custom level file parsing module is configured to parse the custom level file, convert the custom level file into data recognizable by the system, so that the system outputs the corresponding monophonic interactive body according to the custom file.

[0127] The custom level file needs to be parsed by the module before it can be recognized by the system. It can be understood as converting a high-level language into low-level instructions that can be read by the system. The system calls the monophonic interactive body material in the art asset module according to the instructions, and outputs it to the screen in a set manner, and has the expected interaction logic to wait for the user to interact.

[0128] A standard interface module is used to provide a program interface for the custom level file, and the custom level file is generated by an external program.

[0129] The standard interface module serves as an API (application program interface) of the system, and an external program can be connected to generate and edit the custom level file. Of course, the system can also have an editing module for custom levels built-in, and the application is not limited in this regard.

[0130] The numerical management module includes:

[0131] The level metadata module is used to read the numerical information of the single-tone interactive body and the runtime data in the custom level file.

[0132] The numerical information of the single-tone interactive body includes a score, a multiplier, and a musical instrument type. The runtime data of the level includes a level duration and a number of single-tone interactive bodies. After the numerical information is read, the score of each single-tone interactive body can be calculated based on the interactive result, and then the scores of all single-tone interactive bodies are added to obtain the score of the entire level.

[0133] The score calculation module is used to calculate the user score based on the numerical information of the single-tone interactive body and the interactive result between the user and the single-tone interactive body.

[0134] In this embodiment, the formula is:

[0135] ;

[0136] ;

[0137] Where S(N) is the final score value of the current note, s(N) is the preset score, b(N) is the basic score of the current note, acc(N) is the accuracy, 1 for successful interaction and 0 for failed interaction; M f (N) is the note score multiplier, determined by the level creator, and the multiplier of heavy notes is usually higher; m(C) is the continuous hit bonus multiplier, and C is the number of currently continuous hits.

[0138] The continuous hit number calculation module is used to record the current continuous hit number of the user and pass the continuous hit number to the score calculation module as a parameter for calculating the user score. The continuous hit number is the number of successful interactions between the user and the single-tone interactive body within a threshold time interval.

[0139] In order to improve the interest and at the same time give positive feedback to the user for successful performance, thereby promoting the user to learn and understand the corresponding musical instrument in depth, the embodiment also formulates the strategy of "hit" to improve the score, for example, in fast-paced music, the continuous successful interaction to the threshold number of single note interaction bodies triggers "hit", thereby improving the score. For example, 5 single note interaction bodies are successfully interacted in succession, and the score of the 5 single note interaction bodies is doubled.

[0140] The performance progress module is used for calculating the current performance progress and displaying the user score after the performance is completed.

[0141] The sub-module is used for managing the progress of the entire level, and the score calculated by the score calculation module is displayed on the screen after the level is completed.

[0142] Embodiment 2: as shown in the embodiment, the application also provides a virtual reality-based music teaching interaction method, which comprises: Figure 6

[0143] S1 simulates the notes in the music through single note interaction bodies corresponding to the notes, and the single note interaction bodies can visually guide the user to interact;

[0144] S2 The single note interaction body is visually output; when a note in the music is played, the corresponding single note interaction body is displayed according to the preset display time point, pose and expected interaction mode;

[0145] S3 Obtain the interaction result according to the time and mode of the user interacting with the single note interaction body;

[0146] S4 Feedback the interaction result between the user and the single note interaction body.

[0147] In the first mode, the single note interaction body simulated by the single note interaction module is provided with an animation of a musical instrument performance action; when the user interacts with the single note interaction body according to the expected interaction mode, the musical instrument performance action can be performed;

[0148] In the second mode, when the user interacts with the single note interaction body according to the expected interaction mode, the corresponding musical instrument performance action can be performed; the corresponding virtual audience is displayed according to the interaction result of the user and the single note interaction body; and the single note interaction bodies of different musical instruments are interspersed in the same music.

[0149] In the third mode, the single note interaction bodies appearing at the same time are single note interaction bodies of two different musical instruments, and when the user interacts with the single note interaction bodies according to the expected interaction mode, the two different musical instrument performance actions are performed by the two hands respectively.

[0150] ​It should be noted that, in this document, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0151] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a plurality of instructions to make a computer terminal (may be mobile phone, computer, server, or network equipment, etc.) execute the method described in various embodiments of the present application.

[0152] The embodiments of the present application are described above in conjunction with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A virtual reality-based music interaction system, characterized by The music interaction system comprises: a single-tone interaction module, configured to simulate a tone by a single-tone interaction body corresponding to a tone in a music piece, the single-tone interaction body being capable of visualizing an interaction guide for a user and simulating a motion of playing a musical instrument; and obtaining an interaction result according to a time and a manner in which the user interacts with the single-tone interaction body, the interaction result being a successful interaction if the user interacts with the single-tone interaction body according to an expected time and manner; the interaction result being a failed interaction if the user does not interact with the single-tone interaction body according to the expected time and / or manner, the appearance of the single-tone interaction body on a screen being synchronized with a tone played in the music piece; a level visualization module, configured to visualize the single-tone interaction body; when a certain tone in the music piece is played, the corresponding single-tone interaction body is displayed according to a preset display time point, pose and expected interaction manner; a numerical value management module, configured to feed back an interaction result between the user and the single-tone interaction body; the music interaction system comprises at least one of a first mode, a second mode and a third mode; in the first mode, an animation of a playing motion of a musical instrument is set on the single-tone interaction body simulated by the single-tone interaction module; when the user interacts with the single-tone interaction body according to the expected interaction manner, the playing motion of the musical instrument can be performed; in the second mode, when the user interacts with the single-tone interaction body according to the expected interaction manner, a corresponding playing motion of a musical instrument can be performed; a corresponding virtual audience is displayed according to the interaction result between the user and the single-tone interaction body, and the number of the virtual audience is determined by an interaction success rate; and single-tone interaction bodies of different musical instruments are interspersed in the same music piece; and playing motions of various musical instruments are simulated based on positions and interaction directions of the single-tone interaction bodies; in the third mode, two single-tone interaction bodies appear for the same tone in the music piece, the two single-tone interaction bodies being single-tone interaction bodies of two different musical instruments; when the user interacts with the single-tone interaction bodies according to the expected interaction manner, playing motions of the two different musical instruments are performed by two hands respectively; the music interaction system further comprises a custom level file; the custom level file comprises single-tone interaction body information and runtime data; the single-tone interaction body information comprises style information, data information and display information, wherein the style information comprises a shape, a texture and an interaction special effect; the data information comprises a score, a musical instrument type and a score multiplier; and the display information comprises a display time point, a pose and an expected interaction manner; and the runtime data comprises a level duration and a number of single-tone interaction bodies; the numerical value management module comprises: a level metadata module, configured to read single-tone interaction body numerical value information and runtime data in the custom level file; ； ； calculating a score; wherein S(N) is the final score value of the current note, s(N) is a preset score value, b(N) is the instrument base score of the current note, acc(N) is the accuracy, and the interaction success is 1 and the interaction failure is 0; M f (N) is the note score multiplier, which is determined by the level creator, and the multiplier of the accented note is usually higher; m(C) is the continuous hit bonus multiplier, and C is the number of continuously hit notes. a score calculation module, configured to calculate a user score according to an interaction result between the user and the single-tone interaction body based on the single-tone interaction body numerical value information; specifically, a formula is used:

2. The virtual reality-based music interaction system of claim 1, wherein a playing progress module, configured to calculate a current playing progress and display the user score after the playing is completed. the single-tone interaction module comprises: an art material setting module, configured to set a single-tone interaction body style according to style information formulated in the custom level file; a numerical value recording module, configured to set a numerical value of the single-tone interaction body according to numerical value information formulated in the custom level file. The art asset module is configured to model and render the single-tone interactive body, and make approach animation and interactive special effects of the single-tone interactive body, thereby forming a single-tone interactive body material library. The interaction determination module is configured to record the time and manner of the user interacting with the single-tone interactive body, and obtain an interaction result according to the time and manner of the interaction. In the case that the user interacts with the single-tone interactive body in the expected time and manner, the interaction result is a successful interaction. In the case that the user does not interact with the single-tone interactive body in the expected time and / or manner, the interaction result is a failed interaction.

3. The virtual reality-based music interaction system of claim 1, wherein The level visualization module comprises: The external file reading module is configured to read an audio file and a corresponding custom level file, and call the dynamic audio module to play the audio file. The audio file contains a piece of music. The custom level file analysis module is configured to analyze the custom level file, convert the custom level file into data recognizable by the system, and enable the system to output corresponding single-tone interactive bodies according to the custom file.

4. The virtual reality-based music interaction system of claim 3, wherein The level visualization module further comprises: The standard interface module is configured to provide a program interface for the custom level file, and generate the custom level file by using an external program.

5. The virtual reality-based music interaction system of claim 1, wherein The numerical value management module further comprises: The hit number calculation module is configured to record the current hit number of the user, and pass the hit number to the score calculation module as a parameter for calculating the score of the user. The hit number is the number of times that the user successfully interacts with the single-tone interactive body within a threshold time interval.

6. A virtual reality-based music teaching interaction method, deployed in the virtual reality-based music interaction system of any one of claims 1-5, characterized in that The method comprises: The single-tone interactive body corresponding to each note in the piece of music is used to simulate the note, the single-tone interactive body can visually guide the user, and the single-tone interactive body can simulate the action of playing a musical instrument. The appearance of the single-tone interactive body on the screen is synchronized with the note played in the piece of music. The single-tone interactive body is visually output. When a note in the piece of music is played, the corresponding single-tone interactive body is displayed according to a preset display time point, pose, and expected interaction manner. The interaction result is obtained according to the time and manner of the user interacting with the single-tone interactive body. If the user interacts with the single-tone interactive body in the expected time and manner, the interaction result is a successful interaction. If the user does not interact with the single-tone interactive body in the expected time and / or manner, the interaction result is a failed interaction. The interaction result between the user and the single-tone interactive body is fed back. The music teaching interaction method comprises at least one of a first mode, a second mode, and a third mode. In the first mode, the single-tone interactive body simulated by the single-tone interactive module is provided with an animation of a musical instrument playing action. When the user interacts with the single-tone interactive body in the expected interaction manner, the musical instrument playing action can be performed. In the second mode, when the user interacts with the single-tone interactive body in the expected interaction manner, the corresponding musical instrument playing action can be performed. According to the interaction result of the user and the single-tone interactive body, a corresponding virtual audience is displayed, and the number of virtual audiences is determined by the interaction success rate. Different single-tone interactive bodies of different musical instruments are interspersed in the same piece of music. The playing of various musical instruments is simulated based on the position and interaction direction of the single-tone interactive body. In the third mode, two single-tone interaction bodies appear for the same note in the music, and the simultaneously appearing single-tone interaction bodies are single-tone interaction bodies of two different musical instruments, and when the user interacts with the single-tone interaction bodies according to the expected interaction mode, the two hands perform playing actions of the two different musical instruments respectively.

Images