Multi-person music ensemble method, system and storage medium in a virtual scene

Abstract

The present invention discloses a multi-person music ensemble method, system, and storage medium in a virtual scene, belonging to the field of virtual reality technology. The method includes: a server establishes an ensemble channel, responds to an ensemble request from a user terminal, generates a music score model and distributes it to all user terminals, the music score model including a music score table and an audio file; the server executes an ensemble start command, and the user terminal plays the audio file to generate an audio timeline; the server collects the user terminal's performance error data for each beat in real time; generates an error reminder on the display interface of the ensemble channel based on the performance error data; the server forwards the performance error data to other user terminals in the same channel that have not been collected, and the other user terminals extract the error type and replace the next beat content of the current beat in the audio file based on the error type and play it; the server executes an ensemble end command and generates an ensemble record table. This method can prevent the ensemble participants from being disturbed by performance errors.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a multi-person music ensemble method, system and storage medium in a virtual scene. Background Art

[0002] Virtual reality (VR) encompasses computer, electronic information, and simulation technologies. Its fundamental implementation relies primarily on computer technology, leveraging and integrating the latest advances in high-tech technologies such as 3D graphics, multimedia, simulation, display, and servo technology. Using computers and other devices, it creates a realistic virtual world with multiple sensory experiences, including three-dimensional vision, touch, and smell. This creates an immersive experience for those in the virtual world. With the continuous development of social productivity and science and technology, the demand for VR technology is growing rapidly across all industries. VR technology has also made significant progress and is gradually becoming a new field of science and technology.

[0003] With the development of virtual reality technology, the application of virtual scenes is becoming more and more extensive. For example, in a virtual ensemble, users in different locations can enter the virtual scene through the Internet to play music together. However, in order to avoid mutual interference, the traditional ensemble method either uses AI robots to replace real people to play musical instruments, giving users a stand-alone experience when playing, or in order to make the user experience real, the real sounds played by all users are played on the same channel. This method will lead to a situation where the performers have different levels. When a performance error occurs, the erroneous sound will affect the entire ensemble effect, causing other users to be disturbed, thereby affecting the user experience. Therefore, the existing ensemble method cannot avoid interference during ensemble while increasing the real experience. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem that the instrument ensemble in the above-mentioned virtual scene cannot avoid ensemble interference while increasing the real experience, and to provide a multi-person music ensemble method, system and storage medium in a virtual scene, which has the advantages of a more realistic ensemble experience, not being easily disturbed by other users' playing errors, and ensuring the ensemble rhythm even in the case of network delay.

[0005] In a first aspect, the present invention achieves the above-mentioned purpose through the following technical solutions: a multi-person music ensemble method in a virtual scene, comprising the following steps:

[0006] The server establishes an ensemble channel in the virtual scene, responds to ensemble requests from users accessing the ensemble channel, generates a music score model and distributes it to all users, the music score model including a music score table and an audio file;

[0007] The server executes an ensemble start command, and the client plays the audio file and generates an audio timeline according to the beat duration;

[0008] The server collects the performance error data of the user terminal at each beat in real time, and generates an error reminder on the display interface of the ensemble channel according to the performance error data;

[0009] The server forwards the playing error data to other user terminals in the same channel that have not been collected. After receiving the playing error data, the other user terminals find the current beat position according to the audio timeline, extract the error type, and replace the next beat content of the current beat in the audio file according to the error type and play the next beat content;

[0010] The server executes the ensemble end command and generates an ensemble record table.

[0011] Preferably, the method in which the user terminal plays the audio file and generates an audio timeline according to the beat duration includes:

[0012] The beat content of the audio file is composed according to the note playing order of the music score. After the audio file is played, the playing start time of the first note is recorded, and the playing start time of all notes is calculated in sequence according to the beat duration of subsequent single notes. The specific calculation formula is as follows:

[0013] Tn=T0+t(n-1) n≥1

[0014] Where Tn is the start time of playing n notes, T0 is the start time of playing the first note, and t(n-1) is the end time of playing n-1 notes.

[0015] Preferably, the error type is used to determine the type of error, which includes note playing errors, note pitch errors, and delayed or early end of beat duration. The performance error data also includes a note timestamp, which is used to locate the position of the beat corresponding to the note on the audio timeline through time.

[0016] Preferably, the method for using the error type to determine the type of error includes:

[0017] The user terminal compares the notes actually played by the user with the corresponding notes in the music score at the same beat. If there is a difference, the error type is determined to be a note playing error.

[0018] The user terminal compares the pitch of the note actually played by the user with the pitch of the corresponding note in the music score at the same beat. If there is a difference, the error type is determined to be a note pitch error.

[0019] The user end compares the actual playing duration of the notes played by the user with the beat duration on the audio timeline. If the actual duration is shorter, the error type is determined to be premature end of the beat; if the actual duration is longer, the error type is determined to be beat delay.

[0020] Preferably, the method of replacing the next beat content of the current beat in the audio file according to the error type and playing the content includes:

[0021] Read the reception time of the performance error data, and find the corresponding note An within the audio timeline according to the reception time;

[0022] Find the note A(n+1) of the next beat in the audio file according to the audio timeline;

[0023] Analyze the error type. If a note is played incorrectly, note A(n+1) is replaced by a note Bn other than the original note A(n+1). If a note pitch is incorrect, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the pitch error. If a beat duration is delayed or ends prematurely, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the beat duration delay or premature end.

[0024] The replaced beat content is played according to the original sound of the note Bn.

[0025] Preferably, the method for generating an error reminder on the display interface of the ensemble channel according to the performance error data includes:

[0026] Set a scrolling bullet screen on the ensemble channel display interface, and set the content of the scrolling bullet screen to the error reminder content;

[0027] When the server receives the error data, a scrolling bullet screen displays the user ID, the beat position where the error occurred, and the error type.

[0028] In a second aspect, the present invention achieves the above-mentioned object through the following technical solutions: a multi-person music ensemble system in a virtual scene, comprising:

[0029] The server is used to execute the ensemble start and end commands;

[0030] The server also includes:

[0031] An ensemble response unit is configured to establish an ensemble channel in a virtual scene, respond to ensemble requests from a user terminal accessing the ensemble channel, generate a music score model and distribute it to all user terminals, wherein the music score model includes a music score table and an audio file;

[0032] a mistake reminder unit, configured to collect, in real time, performance error data of the user terminal at each beat, and generate a mistake reminder on the display interface of the ensemble channel according to the performance error data;

[0033] A recording unit, used for generating an ensemble record sheet after the ensemble ends;

[0034] The client is configured to play the audio file, generate an audio timeline according to the beat duration of the audio file, and upload performance error data to the server;

[0035] The user terminal also includes:

[0036] The error replacement unit is used to extract the error type, replace the next beat content of the current beat in the audio file according to the error type, and play it.

[0037] Preferably, the error reminder unit includes a barrage module, which is used to set a scrolling barrage on the display interface of the ensemble channel, and set the content of the scrolling barrage as the error reminder content; when the server receives the error data, the scrolling barrage displays the user ID, the beat position where the error occurred, and the error type.

[0038] Preferably, the method for replacing the error replacement unit includes:

[0039] Read the reception time of the performance error data, and find the corresponding note An within the audio timeline according to the reception time;

[0040] Find the note A(n+1) of the next beat in the audio file according to the audio timeline;

[0041] Analyze the error type. If a note is played incorrectly, note A(n+1) is replaced by a note Bn other than the original note A(n+1). If a note pitch is incorrect, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the pitch error. If a beat duration is delayed or ends prematurely, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the beat duration delay or premature end.

[0042] The replaced beat content is played according to the original sound of the note Bn.

[0043] In a third aspect, the present invention achieves the above-mentioned purpose through the following technical solution: a storage medium storing a computer program, which, when executed by a processor, implements the multi-person music ensemble method in a virtual scene as described in the first aspect.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows: when playing an ensemble, the user terminal plays an audio file. Even if one user terminal makes a playing error at the current beat and a sound is inconsistent with the original score, other user terminals can normally hear the correct content of the audio file at the beat, without affecting the playing rhythm of other user terminals. The beat where the error occurs will be processed in the next beat, and the playing content of the next beat will be adjusted according to the error type, to ensure that the error that other user terminals should have heard in the current beat is delayed to the next beat, avoiding interference with the current beat performance, and at the same time, can provide users with a more realistic playing experience; in the case of large network delay, the time when each user terminal receives the noise generated by the playing error is also different, and each user only hears the inconsistent sound in the next beat after receiving the error data, so as not to disturb the experience of other users; with the error reminder, the user can intuitively check which user has made a playing error on the display interface, and the playing error is presented in a way that combines hearing and vision, making the user's experience in the virtual scene ensemble more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the multi-person music ensemble method in Example 1 of the present invention.

[0046] Figure 2 Schematic diagram of the audio timeline in Example 1 of the present invention.

[0047] Figure 3 This is a flowchart of replacing the next beat content and playing it in embodiment 1 of the present invention.

[0048] Figure 4 Schematic diagram of a multi-person music ensemble system in Example 2 of the present invention. Implementation Method

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. Example 1

[0050] like Figure 1 As shown, a multi-person music ensemble method in a virtual scene includes the following steps:

[0051] The server establishes an ensemble channel in the virtual scene, responds to the ensemble request of the user terminal accessing the ensemble channel, generates a music score model and distributes it to all the user terminals. All the user terminals enter the same ensemble channel through the network, for example Figure 1In the example, client 1 and client 2 are assigned to the same ensemble channel by the server through an ensemble request. The ensemble request includes the music they want to ensemble. The server generates a corresponding music score model based on the music requested for the ensemble. The music score model includes a music score table and an audio file. The server distributes the music score table and audio file to client 1 and client 2.

[0052] The server executes an ensemble start command, and the user end plays the audio file and generates an audio timeline according to the beat duration. When the ensemble starts, both the user end 1 and the user end 2 play the audio file. After the audio file starts playing, an audio timeline is generated on the user end. The start time point and end time point of any beat can be found according to the audio timeline. The sound of each beat played by the audio file is the original sound of the note played according to the time sequence of the audio timeline. The method for the user end to play the audio file and generate the audio timeline according to the beat duration includes:

[0053] The beat content of the audio file is composed according to the note playing order of the music score. After the audio file is played, the playing start time of the first note is recorded, and the playing start time of all notes is calculated in sequence according to the beat duration of subsequent single notes. The specific calculation formula is as follows:

[0054] Tn=T0+t(n-1) n≥1

[0055] Where Tn is the starting time of playing n notes, T0 is the starting time of playing the first note, and t(n-1) is the ending time of playing n-1 notes. Figure 2 As shown, when the start time of the first beat is recorded as 10:51:10, the start and end times of the subsequent second, third and fourth beats can be calculated. Finally, the Nth beat is the last beat, and its end time is 10:55:26, at which time the ensemble ends.

[0056] The server collects the performance error data of the user terminal at each beat in real time, and generates an error reminder on the display interface of the ensemble channel according to the performance error data; the server forwards the performance error data to other user terminals in the same channel that have not been collected. After receiving the performance error data, the other user terminals extract the error type, replace the next beat content of the current beat in the audio file according to the error type, and play it. When the user of user terminal 1 makes a mistake while playing, user terminal 1 will send the performance error data to the server. After receiving the data, the server performs two operations at the same time. One step is to generate an error reminder on the display interface according to the data, and the other step is to forward it to user terminal 2. After receiving the performance error data of user terminal 1, user terminal 2 finds the content of the current beat of the audio file according to the audio timeline, thereby finding the playback content of the next beat of the audio file, and performs error replacement on the playback content of the next beat. The replaced content is played when the time reaches the next beat time point. In this way, even if the network delay of user terminal 2 is large, user terminal 2 will only change the content of the next beat of the audio file after receiving the performance error data, ensuring that the audio heard by the user at the current beat is correct. In addition, the error reminder on the display interface can also understand in advance that there is an error in the performance of user terminal 1, avoiding interference that affects the normal performance of user terminal 2.

[0057] The server executes the ensemble end command and generates an ensemble record table. The ensemble record table records the performance of user terminal 1 and user terminal 2, including the number of errors made by each user during the performance, the corresponding error type and the wrong beat. This makes it convenient for users to browse and grasp the points where they made mistakes, which helps to improve the user's instrument ensemble skills.

[0058] In Example 1, the error type can be extracted based on the error data. The error type is used to determine the type of error, which includes a note playing error, a note pitch error, and a beat duration delay or early end. The playing error data also includes a note timestamp, which is used to locate the beat corresponding to the note on the audio time axis by time. The note timestamp is the time point when the user terminal 1 determines that the user has made an error. The time point is accurately displayed during the error reminder. The playing error data is data generated by the user terminal after determining that the current user has made a playing error. The user terminal 2 can extract the error type based on the playing error data. The method for using the error type to determine the type of error includes:

[0059] The user terminal compares the notes actually played by the user with the corresponding notes in the music score at the same beat. If there is a difference, the error type is determined to be a note playing error.

[0060] The user terminal compares the pitch of the note actually played by the user with the pitch of the corresponding note in the music score at the same beat. If there is a difference, the error type is determined to be a note pitch error.

[0061] The user end compares the actual playing duration of the notes played by the user with the beat duration on the audio timeline. If the actual duration is shorter, the error type is determined to be premature end of the beat; if the actual duration is longer, the error type is determined to be beat delay.

[0062] like Figure 3 As shown, when the user terminal 2 obtains the error type corresponding to the error type, the note of the next beat of the audio file can be adjusted. The method of replacing the next beat content of the current beat of the audio file according to the error type and playing it includes:

[0063] Step S1, read the reception time of the playing error data, find out the corresponding note An in the audio time axis according to the reception time, if the note An is the last note corresponding to the audio time axis, then end, if the note An is not the last note corresponding to the audio time axis, then execute step S2;

[0064] Step S2: Find the note A(n+1) of the next beat in the audio file according to the audio timeline;

[0065] Analyze the error type. If a note is played incorrectly, note A(n+1) is replaced by a note Bn other than the original note A(n+1). If a note pitch is incorrect, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the pitch error. If a beat duration is delayed or ends prematurely, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the beat duration delay or premature end.

[0066] Step S3: The replaced beat content is played according to the original sound of the note Bn.

[0067] In Example 1, in order to allow other users to visually observe in real time which user has made a performance error during the ensemble, the server provides a reminder by providing an error reminder on a display interface, so that all users can view the reminder on the display interface. The method of generating an error reminder on the display interface of the ensemble channel based on the performance error data includes:

[0068] Set a scrolling bullet screen on the ensemble channel display interface, and set the content of the scrolling bullet screen to the error reminder content;

[0069] When the server receives the error data, a scrolling bullet screen displays the user ID, the beat position where the error occurred, and the error type. Example 2

[0070] like Figure 4 As shown, a multi-person music ensemble system in a virtual scene includes:

[0071] The server 11 is used to execute the ensemble start command and end command;

[0072] The server 11 also includes:

[0073] An ensemble response unit 111 is configured to establish an ensemble channel in a virtual scene, respond to an ensemble request from a user terminal accessing the ensemble channel, generate a music score model and distribute it to all user terminals, the music score model including a music score table and an audio file;

[0074] The error reminder unit 112 is used to collect the user's performance error data at each beat in real time: generate an error reminder on the display interface of the ensemble channel based on the performance error data, and the error reminder unit 112 includes a barrage module, which is used to set a scrolling barrage on the display interface of the ensemble channel, and set the content of the scrolling barrage to the content of the error reminder; when the server receives the error data, the scrolling barrage displays the user ID, the beat position where the error occurred, and the error type.

[0075] The recording unit 113 is used to generate an ensemble record table after the ensemble ends;

[0076] The user terminal 22 is configured to play the audio file, generate an audio timeline according to the beat duration of the audio file, and upload performance error data to the server terminal;

[0077] The user terminal also includes:

[0078] The error replacement unit 221 is configured to extract an error type, replace the next beat content of the current beat in the audio file according to the error type, and play the next beat content. The replacement method of the error replacement unit 221 includes:

[0079] Read the reception time of the performance error data, and find the corresponding note An within the audio timeline according to the reception time;

[0080] Find the note A(n+1) of the next beat in the audio file according to the audio timeline;

[0081] Analyze the error type. If a note is played incorrectly, note A(n+1) is replaced by a note Bn other than the original note A(n+1). If a note pitch is incorrect, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the pitch error. If a beat duration is delayed or ends prematurely, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the beat duration delay or premature end.

[0082] The replaced beat content is played according to the original sound of the note Bn. Example 3

[0083] This embodiment provides a storage medium comprising a program storage area and a data storage area. The program storage area can store an operating system and programs required for instant messaging functions, while the data storage area can store various instant messaging messages and operating instruction sets. A computer program is stored in the program storage area. When executed by a processor, the multi-person music ensemble method in a virtual scene as described in Example 1 is implemented. The processor can include one or more central processing units (CPUs) or a digital processing unit, etc.

[0084] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0085] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A multi-person music ensemble method in a virtual scene, characterized in that: The following steps are involved: The server establishes an ensemble channel in the virtual scene, responds to ensemble requests from users accessing the ensemble channel, generates a music score model and distributes it to all users, the music score model including a music score table and an audio file; The server executes an ensemble start command, and the client plays the audio file and generates an audio timeline according to the beat duration; The server collects the performance error data of the user terminal at each beat in real time, and generates an error reminder on the display interface of the ensemble channel according to the performance error data; The server forwards the playing error data to other user terminals in the same channel that have not been collected. After receiving the playing error data, the other user terminals find the current beat position according to the audio timeline, extract the error type, and replace the next beat content of the current beat in the audio file according to the error type and play the next beat content; The method for replacing the next beat content of the current beat in the audio file according to the error type and playing the content includes: Read the reception time of the performance error data, and find the corresponding note An within the audio timeline according to the reception time; Find the note A(n+1) of the next beat in the audio file according to the audio timeline; Analyze the error type. If a note is played incorrectly, note A(n+1) is replaced by a note Bn other than the original note A(n+1). If a note pitch is incorrect, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the pitch error. If a beat duration is delayed or ends prematurely, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the beat duration delay or premature end. The replaced beat content is played according to the original sound of the note Bn; The server executes the ensemble end command and generates an ensemble record table.

2. The multi-person music ensemble method in a virtual scene according to claim 1, characterized in that: The method for the user end to play the audio file and generate an audio timeline according to the beat duration includes: The beat content of the audio file is composed according to the note playing order of the music score. After the audio file is played, the playing start time of the first note is recorded, and the playing start time of all notes is calculated in sequence according to the beat duration of subsequent single notes. The specific calculation formula is as follows: Tn=T0+t(n-1) n≥1 Where Tn is the start time of playing n notes, T0 is the start time of playing the first note, and t(n-1) is the end time of playing n-1 notes.

3. The multi-person music ensemble method in a virtual scene according to claim 1, characterized in that: The error type is used to determine the type of error, which includes note playing errors, note pitch errors, and delayed or early end of beat duration. The performance error data also includes a note timestamp, which is used to locate the position of the beat corresponding to the note on the audio timeline through time.

4. The multi-person music ensemble method in a virtual scene according to claim 3, characterized in that: The error type is used to determine the type of error that occurs, including: The user terminal compares the notes actually played by the user with the corresponding notes in the music score at the same beat. If there is a difference, the error type is determined to be a note playing error. The user terminal compares the pitch of the note actually played by the user with the pitch of the corresponding note in the music score at the same beat. If there is a difference, the error type is determined to be a note pitch error. The user end compares the actual playing duration of the notes played by the user with the beat duration on the audio timeline. If the actual duration is shorter, the error type is determined to be premature end of the beat; if the actual duration is longer, the error type is determined to be beat delay.

5. The multi-person music ensemble method in a virtual scene according to claim 1, characterized in that: The method for generating a mistake reminder on the display interface of the ensemble channel according to the performance mistake data includes: Set a scrolling bullet screen on the ensemble channel display interface, and set the content of the scrolling bullet screen to the error reminder content; When the server receives the error data, a scrolling bullet screen displays the user ID, the beat position where the error occurred, and the error type.

6. A multi-person music ensemble system in a virtual scene, characterized in that: include: The server is used to execute the ensemble start and end commands; The server also includes: An ensemble response unit is configured to establish an ensemble channel in a virtual scene, respond to ensemble requests from a user terminal accessing the ensemble channel, generate a music score model and distribute it to all user terminals, wherein the music score model includes a music score table and an audio file; a mistake reminder unit, configured to collect, in real time, the performance error data of the user terminal at each beat, and generate a mistake reminder on the display interface of the ensemble channel according to the performance error data; A recording unit, used for generating an ensemble record sheet after the ensemble ends; The client is configured to play the audio file, generate an audio timeline according to the beat duration of the audio file, and upload performance error data to the server; The user terminal also includes: An error replacement unit is used to extract an error type and replace the next beat content of the current beat in the audio timeline according to the error type. The replacement method of the error replacement unit includes: Read the reception time of the performance error data, and find the corresponding note An within the audio timeline according to the reception time; Find the note A(n+1) of the next beat in the audio file according to the audio timeline; Analyze the error type. If a note is played incorrectly, note A(n+1) is replaced by a note Bn other than the original note A(n+1). If a note pitch is incorrect, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the pitch error. If a beat duration is delayed or ends prematurely, the type of note A(n+1) remains unchanged and is replaced by note Bn in synchronization with the beat duration delay or premature end. The replaced beat content is played according to the original sound of the note Bn.

7. The multi-person music ensemble system in a virtual scene according to claim 6, characterized in that: The error reminder unit includes a barrage module, which is used to set a scrolling barrage on the display interface of the ensemble channel, and set the content of the scrolling barrage as the error reminder content; when the server receives the error data, the scrolling barrage displays the user ID, the beat position where the error occurred, and the error type.

8. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the multi-person music ensemble method in a virtual scene as described in any one of claims 1 to 5 is implemented.

Images