Virtual musical instrument playing method and apparatus, device, and medium

Abstract

The application relates to a virtual musical instrument playing method and device, equipment and medium. In response to a music transcription request, transcription music information is acquired to perform music transcription, and a structured score is generated. The structured score comprises pitch information and pitch timing information. The pitch information is determined to correspond to a target pitch playing control in a virtual musical instrument, and based on the pitch timing information, a playing time window of the target pitch playing control is determined, and corresponding pitch playing sequence information is generated. In response to a playing instruction, playing guidance is performed based on the pitch playing sequence information, and in the process of playing guidance, in response to a touch operation instruction acting on the target pitch playing control, a pitch corresponding to the touch operation instruction is played. The application converts audio into virtual musical instrument playing guidance for a player, reduces the playing threshold of the player, and enriches the source of playable content.

Description

Technical Field

[0001] This application relates to the field of game interaction technology, and in particular to a method, apparatus, device, and medium for playing a virtual musical instrument. Background Technology

[0002] With the development of open-world games, more and more games are providing players with virtual instrument playing functions. Players can control virtual characters to play virtual instruments freely in the game scene, or practice according to the system's preset sheet music. In a typical virtual instrument playing scenario, players trigger the corresponding instrument sound effects by clicking or sliding the operation controls displayed on the virtual instrument interface on the terminal device's touch screen, thereby simulating the process of playing a real instrument.

[0003] In existing virtual instrument playing gameplay, the access to playable content is relatively limited. Players typically rely on pre-installed fixed tracks by the developers or improvise using their own musical knowledge and playing skills. For fixed tracks, the quantity and style are limited by the developers' pre-configuration, making it difficult to meet the diverse and personalized needs of different players. While the free play mode allows players to play as they wish, it requires a considerable level of musical literacy, the ability to identify the pitch and timing of notes, and familiarity with the correspondence between various in-game controls and pitch. For ordinary players lacking this expertise, even if they wish to play a familiar piece, the conversion from audio to a playable sequence is difficult, thus limiting the richness and engaging nature of the virtual instrument playing function.

[0004] Therefore, the relevant technologies have failed to effectively solve the problems of limited performance content and high barriers to user participation, resulting in deficiencies in the richness of content and ease of user participation for virtual musical instrument performance functions. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a virtual musical instrument playing method and corresponding devices, equipment, non-volatile readable storage media, and computer program products.

[0006] According to one aspect of this application, a method for playing a virtual musical instrument is provided, comprising: In response to a music transcription request, the system acquires the music transcription information, performs music transcription, and generates a structured score, which includes pitch information and pitch timing information. Determine the target pitch performance control in the virtual musical instrument corresponding to the pitch information, and based on the pitch timing information, determine the performance time window of the target pitch performance control and generate the corresponding pitch performance sequence information; In response to performance commands, the system guides the performance based on the pitch performance sequence information. During the performance guidance process, it responds to touch operation commands applied to the target pitch performance control and plays the pitch corresponding to the touch operation command.

[0007] According to another aspect of this application, a virtual musical instrument playing device is provided, comprising: The music transcription request module is used to respond to music transcription requests, obtain music transcription information, perform music transcription, and generate structured scores, which include pitch information and pitch timing information. The performance sequence generation module is used to determine the target pitch performance control corresponding to the pitch information in the virtual musical instrument, and based on the pitch timing information, determine the performance time window of the target pitch performance control and generate the corresponding pitch performance sequence information. The performance guidance processing module is used to respond to performance commands, provide performance guidance based on pitch performance sequence information, and, during the performance guidance process, respond to touch operation commands applied to the target pitch performance control and play the pitch corresponding to the touch operation command.

[0008] As can be seen from the above embodiments, the beneficial effects of this application are mainly reflected in the following aspects: The technical solution defined in the independent claims of this application obtains transcribed music information in response to a music transcription request, and generates a structured score containing pitch and pitch timing information, providing a solution for the automatic generation of virtual instrument performance content. By triggering a music transcription request and providing transcribed music information, the system can automatically extract melody information and generate a structured score containing pitch and timing information. This allows audio from various sources to potentially be converted into score content playable on virtual instruments, enriching the sources of playable music and meeting the personalized needs of different players for diverse repertoire.

[0009] Secondly, this application achieves automatic transformation from abstract pitch symbol data to specific interface control operation sequences by determining the target pitch performance control corresponding to the pitch information in the virtual musical instrument and determining the performance time window of each target pitch performance control based on pitch timing information, thereby generating pitch performance sequence information. By automatically executing control matching and performance time window determination operations internally, the system helps reduce the workload of manual arrangement for players, lowers the participation threshold for virtual musical instrument performance functions, and enables ordinary players without musical expertise to obtain operation guidance that allows them to directly follow along in performance.

[0010] Furthermore, this application provides performance guidance based on pitch sequence information in response to performance commands. During the guidance process, it responds to touch input commands applied to the target pitch performance control to play the corresponding pitch, directly linking the system-generated performance guidance with the player's real-time touch input. This end-to-end automated processing method, from audio input to performance guidance output, allows players to complete the entire process from music selection to actual performance without possessing professional music transcription knowledge or the ability to manually arrange sheet music, providing a convenient operational path for virtual instrument performance functions. Attached Figure Description

[0011] Figure 1 This is an exemplary network architecture for this application; Figure 2 This is a flowchart illustrating one embodiment of the virtual musical instrument playing method of this application; Figure 3 An example diagram of the layout structure of the virtual musical instrument playing interface in this application; Figure 4 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, specifically regarding the generation of accompaniment when a player's a cappella audio is a transcribed musical audio. Figure 5 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, which involves decomposing transcribed music audio containing multiple voice parts into multiple independent voice track tracks for players to select from. Figure 6 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, which involves automatic detection and simplification of structured musical scores containing out-of-limit chords. Figure 7 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, which visualizes the decomposition of transcribed musical audio into structural segments with clearly defined segment identifiers. Figure 8 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, which generates multiple pitch playing order information based on the same structured musical score and provides players with operation route switching. Figure 9 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, showing the effect transition between a first and a second visual cue on a pitch playing control; Figure 10 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, which involves loading visual resources corresponding to transcribed music audio. Figure 11This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application regarding a performance fault-tolerant mechanism that compares actual touch events with pitch performance sequence information; Figure 12 This is a flowchart illustrating an embodiment of the virtual musical instrument playing method of this application, which involves collecting players' touch time deviation data to adjust transcriptional constraints in the prompt word template based on the results of aggregate analysis. Figure 13 This is a schematic diagram of the virtual musical instrument playing device of this application; Figure 14 This is a schematic diagram of the structure of a virtual musical instrument playing device used in this application. Detailed Implementation

[0012] The technical solution of this application can be widely applied to various network architectures to adapt to different types and scales of game applications. In a network architecture such as... Figure 1 In the typical network architecture shown, the player's terminal device accesses the game service cluster via the network. This cluster consists of multiple game servers 81. A microservice architecture manages and maintains the operational resources of these game servers 81, providing multiple service instances. Each service instance is responsible for a different service. For example, some service instances can be used to maintain the front-end service for the terminal device 80 to access the game, while others can be responsible for the virtual instrument playing gameplay service, providing players with the corresponding virtual instrument playing function. The player's terminal device 80 has a computer program product implemented according to the virtual instrument playing method of this application installed and running, or the terminal device can be connected to a cloud server container, where the computer program product runs. After the computer program product runs, it accesses the game service cluster, allowing the player to control the player character in the game through the terminal device and interact with various service instances of the game, such as engaging in combat with hostile entities.

[0013] Each service instance can maintain a real-time connection with the player's terminal device 80 via the network, handling various events and interactions in the game, such as player character movement, attacks, and virtual instrument playing. The server provides necessary data support to the terminal device to ensure smooth game operation. This data includes, but is not limited to, data related to virtual instrument gameplay. These datasets contain map model data of the player's game map and resource model data of combat scene resources, such as scene buildings, player character models, enemy player models, enemy monster models, and various other neutral or non-biological models, which are used by the terminal device 80 to render and generate corresponding real-time images. Based on the received virtual instrument gameplay data, the terminal device 80 can generate high-quality real-time images and display them in the graphical user interface, presenting the corresponding virtual instrument model and virtual instrument gameplay GUI to the player.

[0014] This application is not only applicable to the gaming experience of single players, but can also be extended to multiplayer online game environments. In multiplayer games, multiple players' (users') terminal devices connect to the game service cluster simultaneously, and each service instance needs to handle interaction requests from multiple players and update the game status in real time.

[0015] In an exemplary application scenario of this application, consider a massively multiplayer online role-playing game (MMORPG) in which players can freely explore a vast open-world map.

[0016] The virtual musical instrument playing method provided in this application can be widely deployed in various virtual environment systems with virtual musical instrument gameplay, and is particularly suitable for application scenarios such as video games, virtual reality experiences, and simulation training platforms. In a typical technical architecture, the execution entity of this application is usually a terminal device running a client application, such as a personal computer, game console, or mobile device. The game engine or application framework running on the terminal device integrates combat gameplay processing middleware and logic processing units, responsible for real-time rendering of virtual scenes, handling user interactions, and managing the playback of combat performance resources. Virtual musical instrument performance resources and related strategy configuration data can be stored locally on the terminal device or dynamically obtained from a remote server as needed. Real-time monitoring, strategy matching, and the virtual musical instrument playing process can be completed locally on the client to ensure low-latency response and a smooth experience.

[0017] In an exemplary virtual instrument performance process, after equipping a virtual instrument in the game world, a player selects a local audio file or records a short a cappella piece as the music transcription information through the music transcription entry on the interface. Upon receiving the music transcription request, the system transcribes the music information, generating a structured score containing specific pitches and corresponding timing information. The system then determines the target pitch performance control for each pitch based on the correspondence between pitch information and pitch performance controls, and calculates the performance time window for each target pitch performance control based on the pitch timing information, generating a set of pitch performance sequence information that defines the triggering timing of each pitch performance control. When the player clicks the "Start Playing" button, the system enters a performance guidance state, continuously tracking the current performance time. Within the performance time window of a target pitch performance control, a primary visual cue is displayed to prompt the player to trigger that target pitch performance control. Simultaneously, a weaker secondary visual cue is displayed on the next target pitch performance control to be triggered, with the visual intensity gradually increasing as the performance time window of that target pitch performance control approaches until it switches to the primary visual cue. During this process, the player triggers the target pitch performance control within the corresponding time window based on the visual cue on the target pitch performance control. The system responds to the touch operation command by playing the instrument sound effect corresponding to the pitch, allowing the player to follow the guidance and fully play the melody corresponding to the transcribed music information.

[0018] The term "system" as used in this application may refer to a collection of software and hardware environments on the game client and / or server side, used to implement various functions of the game. It includes components such as game engine, physics system, collision detection module, special effects rendering module, and audio system. These components work together to process player input, execute game logic, update game state, generate visual and auditory feedback, and provide players with an interactive experience.

[0019] After providing a general overview of the exemplary network architecture and application scenarios of this application, as well as some basic concepts, the following will continue to describe several specific embodiments of this application.

[0020] Please see Figure 2 The virtual musical instrument playing method of this application includes steps S1100 to S1300 in some embodiments, and each step is described below.

[0021] Step S1100: In response to the music transcription request, obtain the music transcription information, perform music transcription, and generate a structured score, which includes pitch information and pitch timing information.

[0022] In the virtual instrument performance process, the system first responds to the music transcription request, obtains the music transcription information and performs music transcription, thus serving as the starting point for the entire music transcription and performance guidance process. It generates a structured score containing pitch information and pitch timing information, providing a data foundation for the subsequent determination of target pitch performance controls and the generation of pitch performance order information.

[0023] A music transcription request is an instruction used to trigger the system to initiate a music transcription operation. The triggering source for a music transcription request can include various methods. In one implementation, the music transcription request is actively triggered by the player through interactive operations on the game interface. This interaction is not limited to a single button click; it can also be initiated through voice commands, gestures, or shortcut keys. When the player operates the music transcription entry point on the virtual instrument playing interface, the system receives the music transcription request, parses the music information to be transcribed, and passes the transcribed music information to the subsequent music transcription processing. In another implementation, the music transcription request can be automatically triggered by the game's storyline. For example, when a player reaches a specific story node in the main or side quest, the system automatically issues a music transcription request based on the quest script, using a preset audio clip associated with the current storyline as the transcription music information, thereby automatically generating a playable piece that matches the atmosphere of the storyline for the player. In yet another implementation, the music transcription request can be triggered by other players through social interaction. For example, in a multiplayer game, a player can share or gift their favorite music to another player. Upon receiving the share message, the receiving client automatically generates a music transcription request, using the received audio as the transcription music information. In another implementation, the music transcription request can be triggered by a specific in-game event. For instance, when a player enters a specific area in an open world, triggers a specific mechanism, or completes a specific challenge, the system automatically issues a music transcription request, using the background music of that area or the event-related sound effects as the transcription music information.

[0024] Transcribed music information refers to the target audio data to be transcribed, serving as the fundamental object for music transcription. The sources of transcribed music information can include various methods. In some implementations, players can import audio files stored locally on their terminal device into the game client, such as selecting an MP3, WAV, or AAC audio file from the device's music library as the transcription music information. In other implementations, players can also obtain the audio resources to be transcribed through the in-game online audio search function. The system provides an online music library search interface, allowing players to retrieve the corresponding audio stream data from the cloud music library by entering the song name or keywords. Transcribed music information can also be a cappella audio recorded in real-time by the player using the terminal device's audio capture device. After the player operates the recording controls in the performance interface, the system uses audio capture hardware such as a microphone to capture the player's vocal signal. After preprocessing such as noise reduction and resampling, a cappella audio data usable for subsequent music transcription is formed. When the transcribed music information is the player's a cappella audio, the system can also perform chord analysis on the player's a cappella audio, extract chord progression information, and generate an accompaniment track based on the chord progression information and rhythmic features. The accompaniment track is then combined into an accompaniment audio, so that the accompaniment audio and the player's a cappella audio can be played synchronously during the performance guidance.

[0025] Music transcription is the process of converting transcribed musical information into structured musical scores. In practice, music transcription can be achieved through various technical approaches, which are described below.

[0026] In the first implementation, the system employs a pre-trained large language model combined with a cue word constraint mechanism to perform music transcription. In this approach, the system acquires performance configuration parameters, which are player preference settings used to control music transcription and performance guidance effects, including at least one of performance difficulty and genre preference. These performance configuration parameters can be actively set by the player through conditional selection controls on the performance interface, or they can be automatically determined by the system based on the player's historical performance data or default settings.

[0027] The system translates performance configuration parameters into specific transcription constraints. Transcription constraints are a set of parameters used to limit and guide the output of the large language model, including at least one of output format constraints, temporal quantization constraints, target range constraints, and voice part constraints. Output format constraints specify the data representation of the structured score, such as requiring the model to output the pitch, start time, and duration of each note in JSON format. Temporal quantization constraints specify the precision granularity of note timing information, such as requiring the model to quantize the start time of all notes to an eighth-note or sixteenth-note grid. Target range constraints limit the pitch range of the output notes; notes outside this range are automatically transposed an octave into it. Voice part constraints specify the voice part type targeted for transcription, such as requiring the model to extract only the melody part from the audio, ignoring accompaniment or percussion parts.

[0028] The conversion of performance configuration parameters into transcription constraints relies on preset mapping rules. The system maintains an internal constraint rule library, storing combinations of transcription constraints corresponding to different performance difficulties and stylistic preferences. Based on the acquired performance configuration parameters, the system retrieves the corresponding transcription constraints from the library and fills these constraints into the corresponding slots of a preset prompt template, generating the score prompt text. The prompt template is a pre-built text framework containing fixed text and fillable slots. The fixed text describes the task to be performed by the large language model, such as defining the model's role, clarifying the task objective, and specifying the basic requirements for the output format. The fillable slots correspond to different dimensions of the transcription constraints, such as time quantization precision and target pitch range.

[0029] The prompt template is a pre-built text framework containing fixed text and fillable slots. The fixed text describes the task to be performed by the large language model, such as defining the model's role, clarifying the task objective, and specifying the basic requirements for the output format. The fillable slots correspond to different dimensions of transcriptional constraints, such as temporal quantization precision and target pitch range. When generating the music score prompt text, the system will fill the corresponding slots one by one according to the transcriptional constraints found by the mapping rules, forming a complete natural language instruction.

[0030] In an example business scenario, following the scenario described above where a player equips a virtual guqin in the town square and performs "free transcription," the player has set the performance difficulty to "beginner," the music style preference to "pop," and selected an audio file of a pop song as the transcription audio. After obtaining these performance configuration parameters, the system, according to preset mapping rules, sets the time quantization constraint for the "beginner" performance difficulty to an eighth-note grid, the voice part restriction constraint to melody only, and the target range constraint to C3 to C5; the output format constraint for the "pop" music style preference is JSON format. The system fills these transcription constraints into the corresponding slots of the preset prompt word template, generating the sheet music prompt text as shown below: [System Command] You are a professional automatic music transcription system. Your task is to extract melodic note information from a given audio file and output a structured score that meets all of the following constraints.

[0031] [Transcriptional Constraints] 1. Output format: Please output in JSON array format. Each element in the array represents a note. Each note contains the following fields: pitch (MIDI pitch number, integer, range 0 to 127), onset (start time, in milliseconds, integer), and duration (duration, in milliseconds, integer).

[0032] 2. Time Quantization: Quantize the start time and duration of all notes to an eighth-note grid, based on the beat detected in the current audio.

[0033] 3. Target range: Please only output notes with MIDI pitch numbers in the range of 48 (corresponding to C3) to 72 (corresponding to C5). For notes whose original pitch exceeds this range, automatically transpose them by an octave to make them fall into this range.

[0034] 4. Voice Part Restriction: Extract only the main melody from the audio, ignoring all accompanying instruments, percussion, and background vocals. The output should be a monophonic sequence, excluding simultaneous chords.

[0035] [Enter audio] <audio_features> (This refers to audio feature data extracted from transcribed music audio, such as a Mel spectrogram sequence or an embedding vector sequence output by an audio encoder.) < / audio_features> [Output Requirements] Please strictly follow the above constraints to generate structured musical scores, outputting a JSON array directly without any explanatory text.

[0036] The score prompt text structurally comprises four parts. The first part is the system instructions, defining the role and tasks to be performed by the large language model. The second part contains transcription constraints, listing the specific details of output format constraints, time quantization constraints, target range constraints, and voice part restrictions in a numbered list. The time quantization constraint is filled with "eighth note grid," the target range constraint with "MIDI pitch number in the range of 48 to 72," and the voice part restriction constraint with "extract only the melody voice." The third part is the input audio, marked with tags...<audio_features> and< / audio_features> The enclosed area represents the placeholder for audio feature data extracted from the transcribed music audio. The fourth part specifies the output requirements, further standardizing the model's output format. It requires a direct output of a JSON array, prohibiting the addition of explanatory text. This score prompt text, together with the audio feature data from the transcribed music audio, constitutes the complete input to the large language model, from which the model generates a structured score that meets all constraints.

[0037] The system inputs the musical score prompt text and the corresponding audio feature data of the transcribed music information into a pre-trained large language model. The large language model is a generative model based on deep neural networks, typically employing a Transformer architecture. Pre-trained on massive amounts of multimodal data, it possesses the ability to understand natural language instructions and process audio features. During inference, the large language model uses a cross-attention mechanism to apply the constraints in the text instructions to the parsing process of the audio feature sequence. Based on output format constraints, it outputs note information in a specified data format; based on time quantization constraints, it normalizes the start time and duration of notes to a preset beat grid; based on target range constraints, it outputs only notes within a specific pitch range and transposes those outside the range by an octave; and based on voice part constraints, it extracts only the melody of a specified voice part while ignoring other voice parts, ultimately generating a structured musical score that conforms to all constraints.

[0038] After generating the musical score hint text, the system will automatically convert audio data into structured musical score using the hint text and a large language model. The system takes the previously generated musical score hint text and the transcribed music audio provided by the player as input data and submits them to a pre-trained large language model. The model then parses the audio according to the constraints in the hint text and generates a structured musical score containing at least one pitch and corresponding temporal information.

[0039] Large language models are generative models based on deep neural networks, typically employing the Transformer architecture. Pre-trained on massive amounts of text, audio, and multimodal data, they possess the ability to understand natural language instructions and process audio features. In this processing stage, the large language model receives two parts as input: one part is the musical score prompt text, which describes the specific requirements and constraints of the transcription task in natural language; the other part is the transcribed music audio, provided to the model in the form of raw waveforms, acoustic feature sequences, or embedded vector sequences processed by an audio encoder. Internally, the model uses a cross-attention mechanism to apply the constraints in the text instructions to the parsing process of the audio feature sequence, thereby extracting the required note information from the audio.

[0040] During inference, the large language model controls the transcription process based on constraints in the musical score prompts. For example, output format constraints instruct the model to output note information in a specified data format; time quantization constraints instruct the model to normalize the start time and duration of notes to a preset beat grid; target range constraints instruct the model to output only notes within a specific pitch range and transpose notes outside the range by an octave; and voice part constraints instruct the model to extract only the melody of a specified voice part while ignoring other voice parts. By analyzing the time-frequency characteristics of the audio frame by frame, the model identifies the pitch and rhythm information contained therein and performs filtering, quantization, and formatting according to the constraints, ultimately generating a structured musical score that conforms to all constraints.

[0041] In another implementation, the system employs a combination of signal processing and traditional machine learning for music transcription. First, the system performs a Fourier transform or Q-transform on the transcribed music information, converting the audio from a time-domain signal to a time-frequency domain representation, and extracting spectral or chromatographic features. Subsequently, the system uses a pre-trained classification or probabilistic model to estimate the probability of pitch existence at each time frame, and smooths and post-processes the pitch sequence using temporal modeling methods such as Hidden Markov Models or Dynamic Time Warping to determine the start and end times of each pitch, generating a structured score containing pitch information and pitch temporal information.

[0042] Structured musical notation is a data set that includes pitch information and pitch timing information. Pitch information is used to represent the pitch identifiers corresponding to each pitch in the structured musical notation. In the field of music technology, there are multiple ways to represent pitch identifiers. One method is the solfège syllable representation, which names each pitch with identifiers such as Do, Re, Mi, Fa, Sol, La, and Si. This is a commonly used identifier system in music education, facilitating intuitive understanding and singing. Another method is the pitch numbering representation (such as MIDI pitch numbers), a standardized numerical representation used by computers to process music data. It maps each audible pitch to an integer. In virtual instrument performance interfaces, multiple pitch playing controls display solfège syllables such as Do, Re, Mi, Fa, Sol, La, and Si to allow players to intuitively identify the pitch corresponding to each control, reducing cognitive burden. In structured musical notation, pitch information can be stored either in the form of pitch numbers or in the form of solfège syllables. When pitch information uses pitch numbers, the system maintains an internal mapping table between pitch numbers, solfège syllables, and specific pitch performance controls. This ensures that pitch information in the structured score can be accurately located on the corresponding target pitch performance control in the performance interface. Pitch timing information includes the start time and duration of each pitch. The start time indicates the specific position of the pitch on the music's timeline, and the duration indicates the length of time the pitch should be held from the start time. Both can be expressed in milliseconds, sampling frames, or beat positions. In practical engineering implementations, this structured score can be represented as a JSON array, an XML document, or other custom-formatted data structures.

[0043] In one embodiment, the player selects "Real-time Recording" as the source for transcribing music information in the virtual instrument playing interface. The system calls the microphone hardware interface of the terminal device to capture a melody of a popular song sung by the player. After noise reduction and resampling, the player's a cappella audio is generated. The system first performs chord analysis on the player's a cappella audio, converting the audio into a chromatogram sequence through a constant Q-transform. Then, a pre-trained deep neural network model classifies the chromatogram sequence frame by frame, extracting the chord progression information hidden in the a cappella melody, such as identifying the first 8 seconds as a C major triad and the last 8 seconds as a G major triad. Simultaneously, the system performs beat detection on the player's a cappella audio, extracting rhythmic features such as tempo and beat position. Based on the chord progression information and rhythmic features, the system uses a decomposed chord texture template to generate a piano-timbre accompaniment track. This accompaniment track contains an arpeggio-type accompaniment note sequence synchronized with the beat of the player's a cappella audio. The system renders this accompaniment track into audio waveform data through a MIDI synthesis engine and synthesizes it into accompaniment audio. When a player clicks the "Start Playing" control to enter the performance guidance state, the system provides visual guidance to the target pitch playing control based on the pitch performance order information, while simultaneously playing the accompaniment audio and the player's a cappella audio at the same time reference, allowing the player to complete the virtual instrument performance with the combined effect of the a cappella vocals and the system-generated accompaniment.

[0044] In one embodiment, after generating the structured score, the system also performs music structure analysis on the transcribed music information. The system inputs the audio features corresponding to the transcribed music information into a pre-trained music structure analysis model. This model automatically identifies the boundaries and types of different musical segments in the audio by analyzing the repetition patterns and energy contrast features of the audio over time. For example, the system identifies seconds 0 to 15 of the transcribed music information as the "intro," seconds 15 to 45 as the "verse," and seconds 45 to 75 as the "chorus," and generates corresponding segment label sequences. The system aligns these segment label sequences with the performance time windows of each target pitch performance control in the pitch performance sequence information along the timeline. Once the player clicks the "Start Playing" control to enter the performance guidance state, the system provides visual cues to guide the player to the target pitch based on the pitch performance order information. Simultaneously, it continuously monitors the current performance time, comparing it with the start and end times of each segment in the segment label sequence to determine the segment the current performance time falls into. The system then displays the corresponding segment label in real-time in the top information bar of the virtual instrument performance interface, such as "Current Segment: Chorus." When the performance time moves to the next segment, the system updates the displayed content with the corresponding segment label, allowing the player to clearly perceive the structural position of the current performance content within the music while following the performance guidance.

[0045] In one embodiment, the system generates multiple pitch performance order information based on the same structured musical score, with each pitch performance order corresponding to an operation route. Since the virtual instrument performance interface contains multiple octave scale control groups, and the same solfège syllable has corresponding pitch performance controls in the low, middle, and high registers, the system can employ different control matching strategies to match each pitch in the structured musical score to pitch performance controls with the same solfège syllable in different octaves, thus forming different operation routes. For example, the first operation route prioritizes matching pitches to pitch performance controls in the middle register to reduce finger movement on the interface; the second operation route prioritizes matching pitches to pitch performance controls in the high register to present a brighter performance tone. The system displays an operation route switching control on the virtual instrument performance interface, which displays identification information for each operation route, such as using text labels like "Middle Register Fingering" and "High Register Fingering" to distinguish different operation routes. Players can select and enable a specific operation route by clicking the corresponding option in the operation route switching control. After receiving the player's selection command, the system switches the currently enabled pitch performance order information to the pitch performance order information corresponding to the selected operation route, and the subsequent performance guidance process is executed based on the selected operation route.

[0046] In an example business scenario, a player equips a virtual guqin (a seven-stringed zither) in a town square scene of an open-world game and opens the corresponding performance interface. At the top of this interface is a function button labeled "Free Transcription." The player has previously downloaded and stored the audio file of a popular song on their local device and wants to play the melody of this song on the guqin in the game. After clicking the "Free Transcription" button, an audio source selection menu pops up, offering three options: "Import Local File," "Online Search," and "Real-time Recording." The player selects the "Import Local File" option, and the system immediately calls the file selection interface of the terminal device. The player locates and selects the audio file of the target song in the device's storage directory. After the player completes the audio file selection, they click the "Start Transcription" button on the interface, and the system receives the music transcription request. The system uses the audio file selected by the player as the transcription music information, calls the built-in music transcription model to process the audio, extracts the pitch information and pitch timing information of the melody, and generates a structured score containing specific pitches and corresponding timing information. This structured score is used in subsequent processing steps such as control mapping and generation of pitch performance order information.

[0047] As can be seen from the above description, responding to music transcription requests and acquiring transcribed music information to generate structured scores provides the starting instructions and basic input data for the entire music transcription process. Multiple triggering sources for music transcription requests allow the system to flexibly initiate the music transcription process in various scenarios, such as player-initiated requests, game story-driven requests, social sharing interactions, or in-game event triggers. Multiple sources of transcribed music information allow players to use audio obtained from different channels as performance content, breaking through the limitation of the number of preset tracks. Multiple implementation methods for music transcription provide flexible options for the implementation of this technology in different application scenarios and computing environments. The pitch and pitch timing information in the structured score provides an accurate data foundation for subsequent control mapping and performance guidance, solving the problems of limited channels for obtaining performance content and high participation barriers in existing technologies.

[0048] Step S1200: Determine the target pitch performance control in the virtual instrument corresponding to the pitch in the structured musical score, and determine the performance time window of the target pitch performance control based on the pitch timing information, thereby generating the corresponding pitch performance order information.

[0049] After generating the structured score, it is necessary to establish a correspondence between the pitch information in the structured score and the specific pitch performance controls in the virtual instrument performance interface, and based on this, generate pitch performance sequence information to drive the subsequent performance guidance process. The system first extracts each pitch information and its corresponding pitch timing information from the structured score item by item. Then, based on the pitch mapping relationship of each pitch performance control in the virtual instrument performance interface, it determines the target pitch performance control corresponding to each pitch information in the virtual instrument. Finally, it allocates performance time windows to the relevant target pitch performance controls based on the pitch timing information of the relevant pitch information, and finally generates a complete set of pitch performance sequence information that defines the triggering time of each pitch performance control.

[0050] Please refer to Figure 3 , Figure 3The virtual musical instrument playing interface layout shown features 21 pitch control elements arranged in 3 rows and 7 columns in the main interactive area. These 21 pitch control elements correspond to the pitches of 7 different keys within 3 octaves. Each key's pitch control element displays its solfège syllable (Do, Re, Mi, Fa, Sol, La, Si). Pitch control elements with the same solfège syllable across different octaves are distinguished by color or brightness differences: lower register pitch control elements are displayed in a darker tone, middle register elements at standard brightness, and higher register elements in a brighter tone. Below the main interactive area of ​​the virtual musical instrument playing interface are pitch shift controls, used to adjust the overall pitch mapping range of the 21 pitch control elements upwards or downwards. For example, when the pitch-up control is triggered once, the control group that originally covered the middle to high register scales moves up one octave, so that the lowest row of pitch playing controls changes from the low register scale to the middle register scale, and the highest row of pitch playing controls changes from the high register scale to a scale an octave higher.

[0051] Of course, those skilled in the art can flexibly design the specific layout and control styles of the virtual musical instrument playing interface, and are not limited to... Figure 3 The specific forms shown are as follows. For example, in another embodiment, the pitch playing controls can be arranged in a 4x6 matrix, totaling 24 pitch playing controls, covering the pitches of 6 tones in 4 octaves. In yet another embodiment, the pitch playing controls can be arranged horizontally in a single row at the bottom of the screen, and different octave control groups can be switched by swiping left and right. In yet another embodiment, when the virtual instrument model is a piano, the pitch playing controls can be arranged in a keyboard pattern of alternating black and white keys, and the pitch-raising and pitch-lowering controls can be set at both ends of the keyboard. In another embodiment, the timbre switching control can provide more than two timbre options, such as normal tone, harmonics, pizzicato, glissando, and other timbre modes. Regardless of the specific layout of the performance interface, the performance interface includes interactive controls such as pitch playing controls, pitch-raising controls, pitch-lowering controls, and timbre switching controls. Their layout structure and control composition together constitute the interactive basis for virtual instrument performance, providing specific control mapping basis and audio response mechanism for the subsequent generation of pitch playing sequence information and the execution of performance guidance.

[0052] Pitch information in structured musical scores can be represented using solfège syllables or pitch numbers. When pitch information in structured musical scores is represented using pitch numbers, the system maintains a mapping table between pitch numbers, solfège syllables, and specific pitch performance controls. Taking MIDI pitch numbers as an example, number 60 corresponds to the solfège syllable Do, number 62 to Re, number 64 to Mi, and so on. The mapping table also records the interface position of the specific pitch performance control corresponding to each solfège syllable in different octaves. By querying the mapping table, the system can locate each pitch information in the structured musical score. Figure 3 On the corresponding target pitch playing control in the interface.

[0053] In determining the target pitch performance controls, the system needs to match the specific values ​​of pitch information in the structured score with the control layout of the current virtual instrument performance interface. For each pitch information in the structured score, the system first determines which octave the relevant pitch information belongs to and which solfège syllable it corresponds to, and then... Figure 3 Of the 21 pitch playing controls presented, the control that matches the relevant octave scale and solfège syllables is identified as the target pitch playing control. If the player has previously manipulated a key shift control, causing a shift in the overall pitch mapping range of the pitch playing controls, the system will match based on the shifted mapping. For example, if the current control group shifts up an octave due to a key shift control, the control row that originally corresponded to C3 to B3 now corresponds to C4 to B4, and the system will determine the target pitch playing control based on this new mapping.

[0054] When determining the performance time window for the target pitch performance control, the system relies on the pitch timing information associated with the relevant pitch information in the structured score. This pitch timing information includes the start time and duration. The start time determines when the relevant target pitch performance control should be triggered, and the duration determines how long the trigger of the relevant target pitch performance control should be maintained. The system uses the start time as the starting boundary of the performance time window and the sum of the start time and duration as the ending boundary of the performance time window, thereby determining the performance time window corresponding to the relevant target pitch performance control.

[0055] The system performs the aforementioned control matching and performance time window determination operations on each pitch information in the structured score, arranging and integrating all target pitch performance controls and their corresponding performance time windows in chronological order of start time to generate pitch performance sequence information. This pitch performance sequence information is a time-ordered sequence of operation instructions that defines the triggering timing of each target pitch performance control during performance, providing driving data for subsequent performance guidance execution.

[0056] In an example business scenario, consider the scenario described above where a player equips a virtual guqin in the town square and obtains a structured musical score through "free transcription." Assume this structured musical score contains the following simplified melody fragments: the first pitch information is solfège Do, with a start time of 1000 milliseconds and a duration of 500 milliseconds; the second pitch information is solfège Re, with a start time of 1500 milliseconds and a duration of 500 milliseconds; the third pitch information is solfège Mi, with a start time of 2000 milliseconds and a duration of 500 milliseconds, and all of these notes belong to the middle register. Here, the start time represents the specific position of the note on the music's timeline, i.e., the offset from the beginning of the music, rather than a duration. The system first extracts the pitch information of the first note, i.e., the middle register Do, and then... Figure 3 In the interface, the middle register scale is displayed at standard brightness. The pitch control displaying the solfège syllable "Do" within this scale is identified as the target pitch control. The system uses the 1000th millisecond of the start time of this pitch information as the starting boundary of the performance time window, and the 1500th millisecond of the sum of the start time and duration as the ending boundary, thus determining the performance time window corresponding to the relevant target pitch control. Subsequently, the system extracts the pitch information of the second pitch, namely the middle register Re, and determines the corresponding control and assigns a performance time window in the same way. The third pitch information is processed in the same manner. After all notes have been processed, the system generates a set of pitch performance order information. This pitch performance order information sequentially specifies that the middle register Do control is to be triggered between 1000 and 1500 milliseconds, the middle register Re control is to be triggered between 1500 and 2000 milliseconds, and the middle register Mi control is to be triggered between 2000 and 2500 milliseconds.

[0057] As explained above, identifying the target pitch performance controls corresponding to the pitch information in the structured musical score and generating pitch performance sequence information based on pitch timing information achieves the automatic conversion from abstract pitch symbol data to specific interface control operation sequences. The solfège syllables displayed on the pitch performance controls in the virtual instrument performance interface allow players to intuitively identify each control, while the mapping table maintained internally by the system between pitch numbers, solfège syllables, and controls ensures the accuracy of the conversion process. This processing step connects the intelligent transcription results from the front end with the visual performance guidance from the back end, enabling the structured musical score generated by automatic music transcription to be directly converted into actionable instructions for players, solving the cumbersome problem of requiring players to manually match the musical score with controls in existing technologies.

[0058] In step S1300, in response to the performance instruction, performance guidance is performed based on the pitch performance sequence information, and during the performance guidance process, the touch operation instruction acting on the target pitch performance control is responded to, and the pitch corresponding to the touch operation instruction is played.

[0059] After generating the pitch performance order information, the system responds to the performance command, starts the performance guidance process, and performs the performance guidance on the virtual instrument performance interface based on the previously generated pitch performance order information. During the performance guidance process, it continuously responds to touch operation commands applied to the target pitch performance control and plays the pitch corresponding to the touch operation command.

[0060] Performance commands are control signals that trigger the system to initiate the performance guidance process. In practice, performance commands can be triggered by a performance control set on the virtual instrument performance interface. This control may be presented as a button with text labels such as "Start Playing" or "Start Follow-Along." After the player clicks this control, the system receives the performance command, immediately initiates the performance guidance process, loads the previously generated pitch sequence information into the performance guidance execution module, and begins continuously tracking the current performance time with the start time of the music as zero. Performance commands can also be automatically triggered by the system after the music transcription is complete, without requiring manual player intervention.

[0061] Performance guidance is the process by which the system provides visual prompts to the player on the virtual instrument performance interface based on the pitch performance order information. The system continuously tracks the current performance time and compares it item by item with the performance time windows of each target pitch performance control in the pitch performance order information. When the current performance time enters the starting boundary of the performance time window of a target pitch performance control, the system displays the first visual prompt effect for that target pitch performance control within that performance time window, indicating to the player that this target pitch performance control should be triggered. When the current performance time exceeds the ending boundary of this performance time window, the system restores the target pitch performance control to its normal visual state.

[0062] During the performance guidance process, the system can also perform musical structure analysis on the transcribed music information, identify multiple sections and generate corresponding section labels. During the execution of the pitch performance order information, the section label of the currently performed section is displayed in the virtual instrument performance interface. The system can also identify the musical style corresponding to the transcribed music information, select visual resources that match the musical style from a set of preset visual resources, and load and display the visual resources during the performance guidance process.

[0063] Please refer to Figure 3 , Figure 3 The virtual instrument performance interface layout shown can provide initial visual cues through increased brightness, color gradients, the appearance of a diffused halo around the target pitch performance control, or dynamic scaling of the control size. For example, if the pitch performance sequence information indicates that the pitch performance control with the "Do" solfège syllable is displayed at standard brightness and is a pending control between 1000 and 1500 milliseconds, the system will trigger it when the current performance time reaches 1000 milliseconds. Figure 3 The relevant pitch performance controls in the interface are highlighted with a golden glow, while a halo animation emanates from the center of the control, expanding outwards. This highlighting effect and halo animation last until 1500 milliseconds, clearly indicating to the player that this pitch performance control is the target to be triggered. After this performance window ends, the system restores the target pitch performance control to its default appearance when there is no interaction.

[0064] During the performance guidance, the system also displays a secondary visual cue effect on the target pitch performance control corresponding to the next performance time window. Figure 3Taking the virtual instrument performance interface shown as an example, when the player is triggering the target pitch performance control displaying the solfège syllable "Do", the next target pitch performance control to be triggered is the one displaying the solfège syllable "Re". Before the current performance time enters the performance time window of the target pitch performance control displaying the solfège syllable "Re", the system displays a second visual cue effect on this target pitch performance control in advance, such as displaying a light golden glow or semi-transparent highlight on the edge of this target pitch performance control. The visual intensity of the second visual cue effect is weaker than that of the first visual cue effect and will not interfere with the player's attention to the target pitch performance control to be triggered. When the system detects that the time difference between the current performance time and the start time of the performance time window of the target pitch performance control displaying the solfège syllable "Re" is less than a preset threshold, the visual intensity of the second visual cue effect gradually increases inversely proportional to the time difference. That is, the smaller the time difference, the closer the visual intensity is to the first visual cue effect, until the time difference is zero, at which point it completely switches to the first visual cue effect. By combining strong prompts for the current target pitch with weak pre-prompts for the next target pitch, players can anticipate finger movement paths and rhythms, reducing operational errors during continuous playing.

[0065] Touch operation commands are input signals received by the system when the player performs touch operations on the target pitch playing controls during the performance guidance. In the specific implementation, the player observes... Figure 3 The visual cues displayed on each pitch performance control within the interface trigger the target pitch performance control via tapping, pressing, or using the controller buttons within the corresponding performance time window. The system responds to this touch operation by retrieving the corresponding audio signal from the virtual instrument's audio signal library for playback. (See reference...) Figure 3 The audio rules shown involve 42 independent audio signals for the virtual instrument, composed of 3 octaves, 7 pitches, and two types of timbres: normal and overtones. When the player triggers the target pitch playing control (displayed with the solfège syllable "Do") in normal mode, the system plays a sampled Do-pitch instrument with the normal timbre timbre; when the player triggers the same target pitch playing control in overtone mode, the system plays a sampled Do-pitch instrument with the overtone timbre timbre.

[0066] The system also controls the number of audio clips triggered simultaneously. (See reference...) Figure 3 The system's audio rules allow players to trigger multiple audio signals simultaneously for concurrent playback. For example, when playing a chord, a player might press multiple pitch control buttons at the same time. The system plays the corresponding audio signal for each pitch control, creating a multi-note concurrent effect. Triggered audio signals naturally decay and disappear during playback, unaffected or truncated by subsequent audio signals, ensuring the complete listening experience for each note.

[0067] The system also supports fine-tuning the pitch or duration of individual audio signals by short-pressing and long-pressing the pitch control. A short press triggers playback of the audio signal for the default duration, while a long press extends the playback duration of the audio signal or produces subtle changes in pitch depending on the duration of the press.

[0068] When guiding the player through the pitch sequence information, the system can also compare the player's actual touch events with the pitch sequence information. If a touch error or omission is detected exceeding a predetermined threshold, the system will automatically pause or rewind the pitch sequence information and display a recovery prompt. If the player makes multiple consecutive touch errors or omissions during the guidance process, the system will automatically pause the guidance, rewind the current playing position to the last correctly touched target pitch control, and display a recovery prompt on that target pitch control. The player can then restart the performance based on the recovery prompt.

[0069] In an example business scenario, consider the scenario described above where a player equips a virtual guqin (a seven-stringed zither) in a town square and obtains structured musical scores and pitch order information through "free transcription." The player... Figure 3 Clicking the performance control in the upper right corner of the virtual instrument performance interface triggers a performance guidance process after the system receives the instruction. Based on the previously generated pitch performance order information, at the 1000th millisecond of the current performance time, the system highlights the Do solfège control with a gold highlight and a diffused halo as a first visual cue. Simultaneously, the following Do solfège control, also with a Re solfège indicator and at standard brightness, displays a faint gold glow as a second visual cue. The player observes the Do solfège control highlighted and, between 1000 and 1500 milliseconds, clicks this control. The system responds to this touch input, playing a sample Do solfège instrument from the audio signal library. Subsequently, the Re solfège control's visual intensity gradually increases as the performance time window approaches, eventually switching to the first visual cue effect. The player continues to follow the cue to play the subsequent notes. When the player simultaneously triggers multiple pitch performance controls, the system plays multiple corresponding audio signals concurrently. Throughout the performance, players trigger the target pitch control sequentially according to the pitch performance order information. The system responds to the touch operation commands one by one and plays the corresponding pitch, enabling players to fully perform the melody corresponding to the transcribed music audio.

[0070] As can be seen from the above explanation, the system responds to performance commands and executes performance guidance based on pitch sequence information. During the performance guidance process, it responds to touch operation commands applied to the target pitch performance controls to play the corresponding pitch, forming a complete closed loop from system background guidance generation to player-interactive performance. The combination of primary and secondary visual cues on each pitch performance control in the virtual instrument performance interface provides players with current operation guidance and anticipation of upcoming operations, reducing reaction delays and operational errors during continuous performance. The system's real-time audio response to touch operation commands and reasonable control over the number of concurrent audio signals ensure the real-time nature of performance feedback and the integrity of auditory effects, enabling ordinary players without musical expertise to successfully complete musical performances by following the system's guidance.

[0071] Based on any embodiment of the method in this application, such as Figure 3 and Figure 4 As shown, the transcription of music audio includes the player's a cappella audio, including steps S1111 to S1113. Each step is explained below.

[0072] S1111 performs chord analysis on the player's a cappella audio and extracts chord progression information.

[0073] When transcribing music audio into a player's a cappella audio, the system first performs chord analysis on the player's a cappella audio. The player's a cappella audio is real-time audio data recorded by the player through the audio acquisition device on the terminal device. This audio data is a mono vocal signal and contains the melody information sung by the player.

[0074] The purpose of chord analysis is to extract implicit chord progression information from a player's a cappella audio. Chord progression information refers to the sequence of chords in a piece of music arranged chronologically, with each chord containing attributes such as the root note and chord type. Chord progression information provides the basic harmonic framework for generating subsequent accompaniment tracks.

[0075] In its implementation, the system's chord analysis of the player's a cappella audio involves multiple processing stages. In the feature extraction stage, the system resamples the player's a cappella audio to a specified sampling rate, converts it to mono format, and performs silence removal and loudness normalization. Subsequently, the system uses constant Q-transform or Mel spectrogram calculation to convert the audio from a time-domain signal to a time-frequency domain representation, and generates a chromatogram sequence based on this. The chromatogram folds and aggregates the spectral energy according to 12 semitone categories, forming a 12-dimensional vector corresponding to each time frame, representing the energy intensity of each pitch level at that moment.

[0076] During the chord recognition stage, the system uses a pre-trained deep neural network model, such as a convolutional recurrent neural network or an audio processing model based on the Transformer architecture, to take the chromatogram sequence as input and output the chord label probability distribution corresponding to each time frame. The chord label space covers a preset set of chord types, including major triads, minor triads, dominant seventh chords, diminished triads, etc. Each chord type corresponds to 12 root notes, forming a classification space containing dozens to hundreds of categories.

[0077] In the post-processing stage, the system applies mean-mode filtering or Hidden Markov Model decoding to the frame-by-frame probability output of the model to eliminate transient false detection jumps and ensure that chord labels are smooth and continuous in time. The system then detects chord label change points in adjacent time frames. When a label changes and the confidence level exceeds a preset threshold, a chord change event is recorded, and all chord change events are arranged in chronological order to generate structured chord progression information.

[0078] S1112, based on chord progression information and rhythmic features of the player's a cappella audio, generate at least one accompaniment track, the accompaniment track containing a sequence of accompaniment notes synchronized with the beat of the player's a cappella audio.

[0079] After extracting chord progression information, the system further generates an accompaniment track based on the chord progression information and the rhythmic features of the player's a cappella audio. Rhythmic features refer to beat-related information extracted from the player's a cappella audio, including tempo, beat position, and note start time.

[0080] The system first performs beat detection on the player's a cappella audio, identifying the tempo from the audio signal, expressed in beats per minute, and the specific position of each beat on the timeline. When the a cappella audio lacks clear rhythmic cues, the system can infer beat information by combining attack detection and energy changes. Players can also manually adjust the tempo or time signature to assist in the detection.

[0081] The generation of accompaniment note sequences can be based on preset rules or pre-trained generative models. In the rule-based implementation, the system generates an accompaniment note sequence that matches the chord type corresponding to the current time position and a preset accompaniment texture template. The accompaniment texture template defines the performance mode and note distribution of the accompaniment. For example, an arpeggio template arranges the notes of a chord in a certain order into an arpeggio sequence, while a block accompaniment template arranges the notes of a chord simultaneously into a synchronously triggered note group. In the generative model-based implementation, the system uses a pre-trained music generation model, taking chord information and rhythmic features as input, to generate a stylized accompaniment note sequence.

[0082] Each accompaniment track can be assigned a different instrument timbre to generate a richly layered accompaniment effect. The generated accompaniment note sequence is synchronized with the beat of the player's a cappella audio; that is, the start time of the accompaniment notes is quantized to the corresponding beat grid position, ensuring that the accompaniment and the player's a cappella rhythm are coordinated.

[0083] S1113, synthesize at least one accompaniment track into accompaniment audio, so as to play the accompaniment audio and the player's a cappella audio simultaneously during the performance guidance.

[0084] After generating the accompaniment track, the system synthesizes the accompaniment track into a backing audio. The synthesis process is the conversion of a sequence of accompaniment notes in MIDI format or a similar format into playable audio waveform data.

[0085] In its implementation, the system can use the game engine's built-in audio synthesizer for real-time synthesis, or it can use a MIDI synthesis engine to render the generated backing track into an offline audio file. It can also call the terminal device's system audio interface for track mixing. When multiple backing tracks exist, the system renders each track as its corresponding audio waveform, then mixes and overlays them according to their respective timelines to generate the final backing audio. During the mixing process, parameters such as volume ratio, panning position, and effects processing for each track can be adjusted to optimize the overall sound quality of the backing audio.

[0086] When a player clicks the start playing control to enter the playing guidance state, the system provides visual guidance to the pitch playing control based on the pitch playing order information, while simultaneously playing the previously generated accompaniment audio and the player's a cappella audio.

[0087] In its implementation, the system aligns the start time of the accompaniment audio with the start time of the performance guide, ensuring that the playback progress of the accompaniment audio is synchronized with the performance time windows of each target pitch performance control in the pitch performance sequence information. The player's a cappella audio is also played synchronously with the same time reference, ensuring that the player's vocal melody, the system-generated accompaniment, and the instrument pitch played by the player by triggering the pitch performance control are coordinated in time.

[0088] The system allows for independent adjustment of the volume ratio between the accompaniment audio and the player's a cappella audio, enabling players to adjust the volume balance between the accompaniment and vocals as needed. During playback, the accompaniment audio and the player's a cappella audio are unaffected by the player's touch operations and continue to play stably along the timeline, while the instrument pitches generated by the player through triggering the pitch playing controls are played in real-time according to the player's actual touch timing.

[0089] As can be seen, when the transcribed music audio is a player's a cappella audio, the system performs chord analysis on the player's a cappella audio and extracts chord progression information. Based on the chord progression information and rhythmic features, it generates an accompaniment track, synthesizes the accompaniment track into an accompaniment audio, and plays the accompaniment audio and the player's a cappella audio simultaneously during the performance guidance. This achieves the automatic conversion from a player's monophonic a cappella singing to a complete musical performance with harmonic accompaniment. This technical solution solves the problem of the lack of adapted accompaniment support after a player sings a cappella in existing technologies. It allows ordinary players without music composition knowledge to obtain a complete performance experience with accompaniment through simple a cappella operations, lowers the creative threshold of virtual instrument performance functions, and enriches the expressive forms of performance content.

[0090] Based on any embodiment of the method in this application, the performance configuration parameters include at least one of performance difficulty and musical style preference, and the transcription constraints include at least one of output format constraints, time quantization constraints, target vocal range constraints, and part restriction constraints. Different performance difficulties or musical style preferences correspond to different combinations of transcription constraints.

[0091] In the virtual instrument performance process, performance configuration parameters are a set of preference options set by the player on the virtual instrument performance interface or adopted by the system default. These parameters control the fineness of music transcription and the complexity of performance guidance. Performance configuration parameters include at least one of performance difficulty and genre preference. Performance difficulty characterizes the player's desired level of performance complexity and transcription fineness, typically categorized into beginner, intermediate, and advanced levels. Different difficulty levels correspond to different score density, rhythmic complexity, and note selection criteria. Beginner difficulty corresponds to fewer notes and simpler rhythms, suitable for beginners; advanced difficulty corresponds to denser notes, including more rhythmic variations and ornaments, suitable for players with some experience. Genre preference characterizes the player's desired music style, such as traditional Chinese style, pop, rock, electronic, folk, etc. Different genre preferences correspond to different transcription focuses; for example, a preference for traditional Chinese style corresponds to a pentatonic scale-based pitch selection strategy, while a preference for electronic style corresponds to a more refined rhythm quantization grid and dynamic information extraction.

[0092] Transcription constraints are a set of parameters used to limit and guide the output behavior of the large language model, including at least one of output format constraints, temporal quantization constraints, target range constraints, and voice part constraints. Output format constraints specify the data representation of the structured score, for example, requiring the large language model to output in JSON array format, where each element represents a note, and each note contains pitch, start time, and duration fields. Through output format constraints, the system can ensure that the structured score generated by the large language model can be directly parsed and processed by the subsequent control mapping module. Temporal quantization constraints specify the precision granularity of note timing information, for example, requiring the large language model to quantize the start time and duration of all notes to an eighth-note grid or a sixteenth-note grid. Temporal quantization constraints make the generated score more rhythmically regular, conforming to preset beat rules, making it easier for players to play at a uniform tempo. Target range constraints limit the pitch range of the output notes; notes outside this range are automatically transposed by the large language model by an octave to fall within the specified range. Target range constraints ensure that the generated structured score matches the range covered by the pitch control in the virtual instrument playing interface, avoiding notes that cannot be played on the virtual instrument. Voice part constraints specify the voice part type targeted for transcription, such as requiring the large language model to extract only the main melody voice part from the audio, ignoring accompaniment or percussion parts. Voice part constraints enable the system to accurately extract note information of the target voice part from complex audio containing multiple voice parts, generating structured scores adapted to single instruments or specific performance modes.

[0093] There is a mapping relationship between performance configuration parameters and transcription constraints; different performance configuration parameter values ​​correspond to different combinations of transcription constraints. The system maintains a constraint rule library, which stores transcription constraint combinations corresponding to different performance difficulties and stylistic preferences. When the performance difficulty is beginner level, the mapping rule specifies that the time quantization constraint is an eighth-note grid or a more relaxed quantization granularity, the voice part restriction constraint is only for the melody, and the target range constraint is a narrower range. When the performance difficulty is advanced level, the mapping rule specifies that the time quantization constraint is a sixteenth-note grid or a finer quantization granularity, the voice part restriction constraint can be relaxed to include some accompaniment voices, and the target range constraint is a wider range. When the stylistic preference is traditional Chinese music, the mapping rule specifies that the target range constraint adapts to the commonly used range of the pentatonic scale, and the voice part restriction constraint is only for the melody. When the stylistic preference is electronic music, the mapping rule specifies that the output format constraint can include extended fields such as dynamics information, and the time quantization constraint is a finer grid. By mapping the performance configuration parameters to transcription constraints, the system transforms the macro-level preference options that players can easily understand into technical constraint parameters that can be precisely executed by the large language model, thereby achieving flexible control over the music transcription process without increasing the player's operational burden.

[0094] As can be seen, the performance configuration parameters are presented as macro-level preference options that are easy for players to understand, while the transcription constraints act as technical parameters that the large language model can precisely execute, affecting the generation process of structured scores. The correspondence between the two, established through preset mapping rules, allows players to simply set their preferences on the virtual instrument performance interface, and the system can automatically determine the corresponding combination of technical constraint parameters, completing the automatic conversion from player preferences to model instructions. This hierarchical parameter design lowers the operational threshold for players while ensuring the professionalism and usability of the output structured scores, enabling players of different skill levels to receive a performance guidance experience that matches their abilities.

[0095] Based on any embodiment of the method in this application, such as Figure 5 As shown, the process of obtaining transcribed music information for music transcription includes steps S1121 and S1122. The following describes each step.

[0096] Step S1121: Perform sound source separation on the transcribed music audio to obtain at least two independent audio tracks.

[0097] In the virtual instrument performance process, when the transcribed music audio is a complex audio containing multiple voices, the system performs two preprocessing operations before inputting the score prompts and the transcribed music audio into the large language model. These two operations are to separate the sound sources of the transcribed music audio and to receive the player's selection command for individual voice tracks so that automatic music transcription is performed only for the selected voices.

[0098] Sound source separation is an audio signal processing technique used to separate independent audio signals corresponding to different sound sources from a mixed audio file. In this embodiment, the transcribed music audio may contain sound information from multiple parts. For example, a complete song typically contains multiple parts such as the main melody vocals, accompanying instrument sounds, percussion rhythms, and background harmonies. If such composite audio containing multiple parts is directly input into a large language model for music transcription, the structured score output by the model will contain note information from all parts. The generated pitch performance order information may be too complex, exceeding the performance capabilities of a single instrument, or resulting in dense and difficult-to-follow touch prompts in the performance guidance interface. Through sound source separation technology, the system can extract different parts from the transcribed music audio into independent tracks, providing players with a basis for selecting transcription targets.

[0099] In practical implementation, sound source separation can be performed using a pre-trained deep neural network model. Such models typically employ convolutional neural networks or Transformer-based encoder-decoder architectures. After training with a large amount of mixed audio and paired data of individual sound sources, they can learn the characteristic differences of different sound sources in the time-frequency domain. The system inputs transcribed music audio into the sound source separation model, which then analyzes the audio frame by frame, identifying the spectral components belonging to different sound sources and reconstructing them into multiple independent audio waveforms. For example, a pop song audio containing vocals, piano accompaniment, and drum rhythms can, after sound source separation processing, obtain three independent audio tracks: a vocal track, a piano track, and a drum track. Each independent audio track is a mono or multi-channel audio signal, containing the complete sound information of the corresponding sound source in the original audio.

[0100] In its implementation, the inference of the sound source separation model can be executed locally on the terminal device, or the audio can be uploaded to a cloud server for processing by the cloud model, which then sends back the separation results. When sound source separation is executed locally, a lightweight version optimized through model quantization and inference can be used to reduce the computational resource consumption on the terminal device. After separation, the system stores each independent audio track separately and generates a corresponding audio track identifier and waveform preview data for each independent audio track, which can then be used in the subsequent audio selection interface.

[0101] Step S1122: Receive the player's selection command for the independent voice track, and automatically transcribe the selected voice.

[0102] After the sound source separation is completed and multiple independent audio tracks are obtained, the system displays an audio selection area to the player on the virtual instrument playing interface. This area lists information about each independent audio track separated from the transcribed music audio, such as the audio track name, waveform preview, or listening controls. Audio track names can use descriptive text, such as "Main Melody Vocals," "Piano Accompaniment," "Drum Rhythm," or "Bass," allowing players to intuitively understand the sound content represented by each independent audio track. The audio selection area also provides a short listening function for each independent audio track, allowing players to listen to the audio content of each track before making a selection, aiding in determining the target audio track to be transcribed.

[0103] Players select individual voice tracks using the conditional selection controls in the voice selection area. The system receives the selection command from the player and determines one or more target voices specified by the player. The selection command includes the voice identifier of the selected individual voice track. After receiving the selection command, the system only uses the selected individual voice track as input data for subsequent automatic music transcription. Unselected individual voice tracks do not participate in subsequent transcription processing. For example, when a player only wants to play the main melody of a song, they can select only the individual voice track corresponding to "Main Melody Vocals" in the voice selection area. The system will then only perform subsequent automatic music transcription on this individual voice track, and the generated pitch performance order information will only contain the note sequence of the main melody. When a player wants to play the piano accompaniment, they can select the individual voice track corresponding to "Piano Accompaniment".

[0104] As can be seen, through the two preprocessing operations described above, the system decomposes the transcribed music audio containing multiple voice parts into multiple independent voice track tracks, and determines the target voice part for transcription based on the player's selection instructions. This processing method allows players to flexibly select the voice part they wish to play from a complex composite audio, avoiding the generation of overly complex pitch performance order information, and ensuring that the generated performance guidance content matches their own playing ability and the performance characteristics of the virtual instrument. At the same time, the application of sound source separation technology also avoids redundant notes generated by transcribing non-target voice parts from the original audio, improving the clarity and playability of the structured score.

[0105] Based on any embodiment of the method in this application, such as Figure 6 As shown, generating a structured musical score includes steps S1131 and S1132, which are explained below.

[0106] Step S1131: Detect whether there are any overlimit chords in the structured score. An overlimit chord is a chord in the structured score whose number of pitches appearing at the same time exceeds the upper limit of the number of operation controls that can be triggered simultaneously by the virtual instrument.

[0107] In generating pitch performance sequence information, in addition to establishing a correspondence between each pitch in the structured score and the pitch performance controls in the virtual instrument performance interface and determining the performance time window, the system also needs to detect and process any out-of-limit chords that may appear in the structured score. This processing includes two stages: first, detecting whether there are out-of-limit chords in the structured score; when out-of-limit chords are found, converting them into playable single-note sequences or arpeggio sequences according to a preset simplification strategy.

[0108] An over-limit chord refers to a chord in a structured score that, at a given time position, contains more pitches than the maximum number of control actions a virtual instrument can trigger simultaneously. In musical terminology, a chord is a combination of three or more pitches sounding at the same time position. In a virtual instrument performance scenario, the system allows players to trigger multiple pitch control actions simultaneously, but this number is limited by a preset upper limit. When the number of pitches contained in a chord at a certain time position in the structured score exceeds this upper limit, the chord is considered an over-limit chord.

[0109] The system scans and detects pitch information in the structured score at each time position. For each time position, the system counts the number of pitches appearing simultaneously at that time position and compares this number with a preset upper limit for the number of simultaneously triggerable operation controls. When the count exceeds the upper limit, the system marks all pitch combinations at that time position as over-limit chords. For example, if the virtual instrument playing interface allows players to trigger a maximum of two pitch playing controls simultaneously, and three pitches appear simultaneously at a certain time position in the structured score, then the combination of these three pitches is an over-limit chord. The existence of over-limit chords means that the player cannot trigger all corresponding pitch playing controls simultaneously with a single operation at the current time position, requiring the system to simplify and convert them into a playable form for the player.

[0110] Step S1132: When there is an overlimit chord, the overlimit chord is converted into a playable single-note sequence or arpeggio sequence according to a preset simplification strategy; wherein, the single-note sequence refers to the sequence generated by retaining the highest note or root note of the overlimit chord, and the arpeggio sequence refers to the sequence generated by decomposing the overlimit chord into arpeggios according to a preset pitch order.

[0111] When the system detects an out-of-limit chord in the structured score, it transforms the out-of-limit chord according to a preset simplification strategy. The simplification strategy defines the specific rules for converting the out-of-limit chord into a playable form, and the result can be a sequence of single notes or an arpeggio sequence.

[0112] A single-note sequence is a sequence formed by selecting only one pitch from multiple pitches of an overlimit chord. This can be achieved by either retaining the highest pitch or the root note of the overlimit chord. Retaining the highest pitch means selecting the note with the largest pitch value from all pitches in the overlimit chord and using it as the sole note in the single-note sequence, discarding the rest. Retaining the root note means selecting the root note from the overlimit chord and using it as the sole note in the single-note sequence. Single-note sequences ensure that only one pitch playing control is triggered at each time position, guaranteeing that the number of triggered controls will not exceed the limit.

[0113] An arpeggio sequence is a sequence formed by sequentially decomposing and arranging the pitches of an overlimit chord according to a preset order. Unlike a single-note sequence, an arpeggio sequence retains multiple pitches in an overlimit chord, but instead of triggering them simultaneously, it triggers them sequentially at certain time intervals. The preset order can be an ascending order from low to high, a descending order from high to low, or other orders determined according to specific music theory rules. The pitches in an arpeggio sequence are assigned extremely short time intervals, allowing players to trigger multiple pitch playing controls sequentially within a short period, simulating the auditory effect of a chord.

[0114] In practice, the specific simplification strategy can be automatically determined by the system based on the performance difficulty in the performance configuration parameters, or it can be manually specified by the player on the virtual instrument performance interface. For example, when the beginner performance difficulty is set, the system prioritizes the simplification method of single-note sequences, retaining only the highest note or root note to reduce the performance difficulty; when the advanced performance difficulty is set, the system can use the simplification method of arpeggio sequences, retaining more chord information to enrich the performance expressiveness.

[0115] As can be seen, the system automatically detects and simplifies any out-of-limit chords that may exist in the structured score. The out-of-limit chord detection mechanism ensures that the system can identify pitch combinations that exceed the performance capabilities of the virtual instrument, avoiding the generation of performance instructions that the player cannot execute. The simplification strategy transforms chords that were originally unplayable into operable forms for the player. While preserving the original musical melody characteristics, it adapts to the operational constraints of the virtual instrument performance interface, enabling scores of any complexity generated by the large language model to be transformed into practically feasible performance instructions.

[0116] Based on any embodiment of the method in this application, such as Figure 7 As shown, music structure analysis is performed on transcribed music information, including steps S1141 and S1142. The following describes each step.

[0117] Step S1141: Before guiding the performance based on the pitch performance sequence information, perform music structure analysis on the transcribed music information, identify multiple performance segments and generate corresponding segment labels.

[0118] Before guiding the performance based on the pitch performance order information corresponding to the structured score, in addition to establishing a correspondence between each pitch in the structured score and the pitch performance control in the virtual instrument performance interface and determining the performance time window, the system can also perform music structure analysis on the transcribed music audio, identify multiple paragraphs and generate corresponding paragraph labels, and display the paragraph labels of the current performance paragraph in real time during the subsequent performance guidance process.

[0119] Music structure analysis refers to the process by which a system analyzes the acoustic characteristics of transcribed music audio data to automatically identify the boundaries of different musical segments and determine the type of each segment. A complete piece of music typically consists of multiple structural segments with different musical characteristics, such as an introduction, verse, chorus, bridge, and coda. Different segments differ in melodic progression, harmonic arrangement, rhythmic density, and timbre configuration. These differences are reflected in the spectral characteristics and energy distribution of the audio signal, and can be detected and identified by the system through algorithms.

[0120] Segment tags are markers generated by the system after performing music structure analysis on the transcribed music audio. These tags are used to identify different musical segments. During the music structure analysis, the system automatically analyzes the acoustic features of the transcribed music audio to identify multiple performance segments, such as intro, verse, chorus, bridge, and outro, and generates a corresponding segment tag for each segment. Each segment tag includes the start and end times of the performance segment, as well as a segment type identifier.

[0121] The function of segment tags is to represent the musical segments within the transcribed music audio that correspond to the current performance progress. During the performance guidance, the system continuously tracks the current performance time and compares it with the start and end times of each segment tag. When the system detects that the current performance time falls within the time range of a certain segment tag, the musical segment represented by that segment tag is the one corresponding to the current performance progress. The system displays the segment type identifier of this segment tag on the virtual instrument performance interface, allowing the player to understand in real time the position of the current performance content within the overall structure of the music.

[0122] In its implementation, the system can employ a pre-trained music structure analysis model to perform segment recognition. This model takes acoustic features extracted from transcribed music audio as input, such as Mel spectrogram sequences, chromatogram sequences, or self-similarity matrices. It analyzes the temporal structural patterns of the audio using convolutional neural networks, recurrent neural networks, or Transformer-based sequence models, outputting segment type labels and segment boundary positions corresponding to each time point. For example, the model might output that 0 milliseconds to 15000 milliseconds is the "intro" segment, 15000 milliseconds to 45000 milliseconds is the "verse A" segment, 45000 milliseconds to 75000 milliseconds is the "chorus B" segment, and so on. Alternatively, the system can use traditional methods based on self-similarity matrix analysis. By calculating the feature similarity between different time frames of the audio, it detects repeated segments and structural change points in the audio, and infers the type label for each segment by combining these with music theory rules.

[0123] After completing paragraph recognition, the system generates a structured sequence of paragraph labels, where each label contains the paragraph's start time, end time, and paragraph type identifier. This sequence of paragraph labels is associated with the pitch performance order information and aligned with the performance time windows of each target pitch performance control on the timeline. By associating paragraph labels with performance time windows, the system can determine the musical paragraph to which each performance time window belongs during the performance guidance process.

[0124] Step S1142: When guiding the performance based on the pitch performance order information, display the segment label of the current performance segment. The segment label is used to represent the music segment in the transcribed music information that corresponds to the current performance progress.

[0125] During the performance guidance process based on pitch performance sequence information, the system displays the segment labels of the current performance segment in real time on the virtual instrument performance interface according to the comparison results between the previously generated segment label sequence and the current performance time.

[0126] Once the player clicks the "Start Playing" control to enter the performance guidance state, the system continuously tracks the current performance time and compares it with the start and end times of each segment in the segment label sequence. When the system detects that the current performance time falls within the time range of a certain segment, it identifies that segment's type as the current segment label and displays it in a designated area of ​​the virtual instrument performance interface. When the current performance time enters the time range of the next segment, the system updates the displayed content to the segment label corresponding to the new segment.

[0127] The paragraph labels on the virtual instrument performance interface can be displayed above the main interactive area, next to the pitch performance control area, or in the top information bar of the performance interface. For example, they can be displayed in... Figure 3 The virtual instrument playing interface shows a text prompt such as "Current Section: Verse A" or "Current Section: Chorus B" in the center of the top area, or it is presented visually in the form of icons, progress bars, etc., so that players can clearly perceive their current position in the musical structure during the performance.

[0128] In a sample business scenario, after performing musical structure analysis on the transcribed music audio from the transcribed music information, the system identifies seconds 0 to 15 as the intro, seconds 15 to 45 as the verse, and seconds 45 to 75 as the chorus, and generates segment labels for each of these three sections. When the player begins playing, and the current playing time advances to 20 seconds, the system detects that 20 seconds falls within the time range corresponding to the verse segment label. This segment label indicates that the musical segment corresponding to the current playing progress is the verse. The system displays the message "Current Segment: Verse" on the virtual instrument playing interface, allowing the player to clearly perceive the structural position of the current playing progress within the music.

[0129] As can be seen, the system automatically breaks down transcribed music information into structural segments with clear segment identifiers and presents the current segment information to the player in real time during the performance guidance. This visual design of music structure allows players to clearly understand the position of the current performance content within the entire piece while following the performance guidance, facilitating segmented practice or focusing on specific sections. It also enhances the sense of structure and direction during the performance, improving the interactivity and learning support value of the virtual instrument performance function.

[0130] Based on any embodiment of the method in this application, such as Figure 8 As shown, the music transcription process generates multiple operation routes for pitch performance order information for players to select and enable, including steps S1151 and S1152. The following describes each step.

[0131] Step S1151: Display the operation route switching control. The operation route switching control is used to display at least two operation routes for the player to select and enable.

[0132] In the process of generating pitch performance order information, in addition to establishing a correspondence between each pitch information in the structured score and the pitch performance control in the virtual instrument performance interface and determining the performance time window, the system can also generate multiple pitch performance order information based on the same structured score and provide players with operation route switching controls for selection.

[0133] After the same structured musical score is matched with controls and the performance time window is determined, multiple sets of different pitch performance order information can be generated. Each pitch performance order information corresponds to an operation route. The operation route refers to the movement path and triggering order of the player's fingers among the various pitch performance controls in the virtual instrument performance interface when playing the same piece of music. Since there may be multiple octave scale control groups in the virtual instrument performance interface, and the pitch information of the same solfège syllable has corresponding pitch performance controls in the low, middle, and high registers, the system can theoretically match the same pitch information in the structured musical score to pitch performance controls with the same solfège syllable in different octave scales, thus forming different operation routes.

[0134] In generating multiple pitch performance order information, the system uses the same structured musical score as input data and employs different control matching strategies. It performs multiple control matching and performance time window determination operations for each pitch information in the structured score, each time using a different matching preference. For example, one matching preference prioritizes matching pitch information to the middle register pitch performance controls to reduce finger movement on the interface; another preference prioritizes matching pitch information to the high register pitch performance controls to make the performance tone brighter; yet another preference dynamically selects the optimal position of the pitch information within adjacent octave controls based on the principle of minimizing finger movement. The system performs complete control matching and performance time window determination operations according to different matching preferences, generating an independent set of pitch performance order information each time, with each pitch performance order information corresponding to a specific operation path.

[0135] Taking the previous scenario where players equipped a virtual guqin in the town square and obtained a structured musical score through "free transcription" as an example, this structured musical score contains three pitch information: middle register Do, middle register Re, and middle register Mi. The system uses a first matching preference, matching the middle register Do to a pitch playing control displaying the Do solfège syllable at standard brightness; the middle register Re to a pitch playing control displaying the Re solfège syllable at standard brightness; and the middle register Mi to a pitch playing control displaying the Mi solfège syllable at standard brightness, generating the first set of pitch playing order information. The system then uses a second matching preference, matching the middle register Do to a pitch playing control in the treble scale that is displayed in a brighter hue and has the Do solfège syllable; the middle register Re to a pitch playing control in the treble scale that is displayed in a brighter hue and has the Re solfège syllable; and the middle register Mi to a pitch playing control in the treble scale that is displayed in a brighter hue and has the Mi solfège syllable, generating the second set of pitch playing order information. The two sets of pitch performance order information correspond to two different operation routes, and players can choose according to their own playing habits and preferences.

[0136] After generating multiple pitch performance sequence information, the system displays an operation path switching control on the virtual instrument performance interface. This control is an interactive feature that allows players to switch between various operation paths and can be displayed on... Figure 3 Above or beside the main interactive area of ​​the virtual musical instrument playing interface shown.

[0137] The operation route switching control displays identification information for each operation route, which may include the route's name, number, thumbnail, or simplified diagram. For example, the first operation route could be labeled "Middle Register Fingering," the second "High Register Fingering," and the third "Comfortable Fingering." Players can select and activate a specific operation route by clicking, touching, or swiping the corresponding option in the operation route switching control. After receiving the player's selection command, the system uses the selected operation route as the final operation route, switching the currently activated pitch performance order information to the pitch performance order information corresponding to the selected operation route. Subsequent performance guidance is then executed based on the pitch performance order information corresponding to this final operation route.

[0138] The options in the operation route switching control can be visually displayed in different selected and unselected states. Options corresponding to the currently enabled operation route are highlighted, bolded, or marked with a special color, while options corresponding to unenabled operation routes appear in their normal visual state, making it easy to distinguish the currently selected option. Players can preview and switch between different operation routes using the operation route switching control before the performance begins, and can also switch routes while the performance is paused.

[0139] Step S1152: Respond to the route selection instruction of the operation route switching control, take the selected operation route corresponding to the route selection instruction as the final operation route, and perform performance guidance based on the pitch performance sequence information corresponding to the final operation route.

[0140] In response to a route selection command from the route switching control, the system designates the selected route as the final route. The route selection command is the selection signal received by the system when the player clicks, touches, or swipes the corresponding option in the route switching control. Upon receiving the command, the system confirms the selected route as the final route and switches the currently active pitch order to the set of pitch order corresponding to the selected route. The options in the route switching control can visually display different selected and unselected states. Options corresponding to the currently selected route are highlighted, bolded, or marked with a special color, while options corresponding to unselected routes display a normal visual state, clearly distinguishing the currently active route. Players can preview and switch between different routes using the route switching control before playing, and can also switch routes while playing is paused.

[0141] The performance guidance is based on the pitch performance sequence information corresponding to the final operation route. This means that in subsequent performance guidance processes, the system uses the pitch performance sequence information corresponding to the selected final operation route as the driving data. Based on the performance time window of each target pitch performance control in this pitch performance sequence information, the system provides visual guidance on the virtual instrument performance interface, enabling the player to complete the performance according to the operation path defined by the selected operation route.

[0142] As explained above, multiple pitch performance sequences are generated based on the same structured musical score, and players are provided with controls to switch between different operation routes. The selected route in these controls becomes the final operation route, allowing the same piece of music to correspond to multiple different operation routes for players to choose from according to their own habits and preferences. This multi-route design solves the problem that a single control mapping method is difficult to adapt to different players' operating habits, improves the personalized adaptability and user-friendliness of virtual instrument performance functions, and enables players with different hand shapes and playing styles to have a comfortable operating experience.

[0143] Based on any embodiment of the method in this application, such as Figure 9 As shown, the performance guidance is based on the pitch performance sequence information, including steps S1311 to S1313. The following is a description of each step.

[0144] Step S1311: Determine the first target pitch performance control corresponding to the performance time window in which the current time falls, and make the first target pitch performance control display the first visual cue effect within the performance time window.

[0145] During the performance guidance process, the system provides players with current operation guidance and prediction of upcoming operations by presenting visual prompts of different intensities on the virtual instrument performance interface for each target pitch performance control.

[0146] The system continuously tracks the current performance time and compares it item by item with the performance time windows of each target pitch performance control in the pitch performance sequence information. When the system detects that the current performance time enters the starting boundary of the performance time window of a target pitch performance control, it identifies that target pitch performance control as the first target pitch performance control and displays the first visual cue effect within that performance time window.

[0147] The first target pitch performance control refers to the target pitch performance control that is currently in a state of pending triggering at the current performance time. The first visual cue effect refers to the display state with the strongest visual appeal applied to the target pitch performance control that should be triggered at the moment, its purpose being to clearly inform the player which pitch performance control should be triggered within the current performance time window. (See reference...) Figure 3 The layout structure of the virtual musical instrument playing interface shown can be visually prompted by increasing the brightness of the first target pitch playing control, changing its color, displaying a diffused halo around it, or dynamically scaling the size of the control.

[0148] Following the aforementioned scenario where players equip a virtual guqin in the town square, when the pitch performance sequence information indicates that a pitch performance control displaying the "Do" solfège syllable and shown at standard brightness is a pending control between 1000 and 1500 milliseconds, the system, at the 1000th millisecond of the current performance time, makes the pitch performance control displaying the "Do" solfège syllable exhibit a golden highlight effect, and simultaneously generates a halo animation that expands outward from the center of the pitch performance control. This golden highlight and expanding halo animation constitute the first visual cue effect, lasting until 1500 milliseconds, clearly indicating to the player during this period that the pitch performance control is the target that should be triggered. After this performance time window ends, the system restores the first target pitch performance control to its default display appearance when there is no interaction.

[0149] Step S1312: Display the second visual cue effect on the second target pitch performance control corresponding to the next performance time window. The visual intensity of the second visual cue effect is weaker than that of the first visual cue effect.

[0150] While the system displays the first visual cue effect for the first target pitch performance control, it also searches for the target pitch performance control corresponding to the next performance time window based on the pitch performance order information, identifies the target pitch performance control as the second target pitch performance control, and displays the second visual cue effect on the second target pitch performance control in advance.

[0151] The second target pitch performance control refers to the next target pitch performance control that will enter the pending trigger state after the performance time window of the first target pitch performance control ends. The second visual cue effect refers to a pre-shown warning with weak visual appeal applied to the upcoming target pitch performance control, the purpose of which is to inform the player in advance of the location of the next pitch performance control that should be triggered, helping the player to predict the movement path and rhythm of their fingers.

[0152] The visual intensity of the second visual cue is set to be weaker than that of the first visual cue to avoid competing with the player's visual attention for the cue provided by the current target pitch playing control. (See reference...) Figure 3 The layout structure of the virtual musical instrument playing interface shown can be presented with a second visual cue effect, which can be represented by a faint golden glow appearing at the edge of the second target pitch playing control, or by the second target pitch playing control being covered with a layer of low-opacity semi-transparent highlight.

[0153] Following the aforementioned business scenario, when a player is preparing to trigger the first target pitch performance control displaying the "Do" solfège syllable, the next target pitch performance control to be triggered is the one displaying the "Re" solfège syllable. The system pre-emptively displays a faint gold-edged glow on the target pitch performance control displaying the "Re" solfège syllable as a secondary visual cue. The visual intensity of this faint gold-edged glow is significantly weaker than the combination of a bright gold highlight and a diffused halo effect on the first target pitch performance control displaying the "Do" solfège syllable, keeping the player's attention focused on the target pitch performance control that should be triggered, while allowing them to perceive the location of the next target pitch performance control in their peripheral vision.

[0154] Step S1313: When the time difference between the current time and the start time of the performance time window of the second target pitch performance control is less than a preset threshold, the visual intensity of the second visual cue effect is gradually increased inversely to the time difference until the time difference is zero, at which point the first visual cue effect is switched.

[0155] The system continuously monitors the time difference between the current performance time and the start time of the performance time window of the second target pitch performance control. When the system detects that this time difference is less than a preset threshold, it begins to dynamically adjust the visual intensity of the second visual cue effect. The preset threshold is a preset time constant used to define the critical moment when the second visual cue effect begins to transition to the first visual cue effect. For example, it can be set to 300 milliseconds, meaning that the transition process starts when there are still 300 milliseconds between the current performance time and the start time of the performance time window of the second target pitch performance control.

[0156] The dynamic adjustment of the visual intensity of the second visual cue effect follows the principle of being inversely proportional to the time difference. That is, the smaller the time difference between the current performance time and the start time of the performance time window, the stronger the visual intensity of the second visual cue effect. When the time difference is zero, that is, when the current performance time exactly enters the start boundary of the performance time window of the second target pitch performance control, the visual intensity of the second visual cue effect reaches its maximum value, and it completely switches to the first visual cue effect.

[0157] The gradual change in visual intensity can be achieved by adjusting one or more visual parameters such as luminance, color transparency, aperture radius, or control scaling. For example, when the time difference is 300 milliseconds, the edge luminance of the second target pitch playing control is about 30% of its initial value; when the time difference is 150 milliseconds, the edge luminance increases to about 60%, and the transparency of the semi-transparent highlight increases accordingly; when the time difference is 50 milliseconds, the luminance is close to 90% of the full highlight state, and the aperture begins to expand outward; when the time difference is zero, the edge luminance completely transforms into a golden highlight, the aperture fully expands, the semi-transparent highlight becomes completely opaque, and the second visual cue effect completely switches to the first visual cue effect. At this point, the original second target pitch playing control becomes the new first target pitch playing control, and the system synchronously begins to present a new second visual cue effect on the next target pitch playing control that follows immediately.

[0158] Following the aforementioned business scenario, when the system detects that the time difference between the current performance time and the 1500th millisecond of the performance time window start time of the target pitch performance control displaying the "Re" solfège syllable identifier is less than the preset 300 milliseconds, the faint gold edge glow originally displayed on the target pitch performance control begins to gradually intensify. As the current performance time approaches the 1500th millisecond, the brightness of the edge glow continues to increase, and the outline of a diffused halo gradually appears around it. When the current performance time reaches the 1500th millisecond, the visual cue effect on the target pitch performance control has completely switched to the first visual cue effect of gold highlight and diffused halo, officially becoming the new first target pitch performance control. At the same time, the system begins to display the second visual cue effect of faint gold glow on the target pitch performance control displaying the "Mi" solfège syllable identifier immediately following behind.

[0159] As can be seen, the virtual musical instrument performance interface combines primary and secondary visual cues on the target pitch performance controls, with a gradual transition from the secondary cues to the primary ones, forming a complete visual guidance system. Strong cues for the current control ensure players can accurately identify the current target, while weak cues for the next control provide advance guidance on finger movement paths. Gradually increasing visual intensity provides players with an intuitive sense of approaching the optimal moment for their action. This hierarchical visual cue design effectively reduces reaction time and operational errors during continuous performance, improving the smoothness and accuracy of players following the performance guidance.

[0160] Based on any embodiment of the method in this application, such as Figure 3 and Figure 10 As shown, the performance guidance is based on the pitch performance sequence information, including steps S1321 and S1322. The following is an explanation of each step.

[0161] Step S1321: Identify the music style corresponding to the transcribed music audio information.

[0162] During the performance guidance process, the system also performs two related operations: music style recognition and visual resource loading. First, it identifies the music style corresponding to the transcribed music information. Second, it selects matching visual resources from multiple preset visual resources based on the identified music style and loads and displays the visual resources during the performance guidance process.

[0163] Music style recognition refers to the process by which a system analyzes the audio features of transcribed music information to automatically determine the music style type to which the audio belongs. Music styles can include various categories such as traditional Chinese music, pop, rock, electronic music, folk, and jazz. In specific implementations, the system can use a pre-trained audio classification model to perform music style recognition. This audio classification model takes acoustic features extracted from the transcribed music information as input, such as Mel spectrogram sequences or embedding vector sequences output by an audio encoder. After layer-by-layer processing through convolutional layers, recurrent layers, or self-attention layers, the output layer generates a confidence probability distribution for each music style category. The system selects the music style category with the highest confidence as the recognition result. The timing of music style recognition can be pre-completed during music transcription or performed in real time when the performance guidance begins.

[0164] Step S1322: Select a visual resource that matches the music style from a set of preset visual resources, and load and display the visual resource during the performance guidance process.

[0165] Visual resources are collections of materials tied to specific music styles, used to decorate and enhance the visual presentation of the virtual instrument playing interface. The system pre-installs multiple sets of visual resources, each associated with one or more music styles. Visual resources include at least one of the following: visual patterns for pitch playing controls and audio waveform effects. The visual patterns for pitch playing controls refer to the texture, color, or border style of each pitch playing control in the virtual instrument playing interface. Different music styles correspond to different visual patterns for pitch playing controls in terms of texture, color, or border style. For example, the pitch playing controls for the traditional Chinese style use a wood grain texture with a dark border, the pitch playing controls for the rock style use a dark metal panel with a red outline, and the pitch playing controls for the electronic style use a neon light strip glow. Audio waveform effects refer to the visual animation of sound waves that appear around the pitch playing controls or in specific areas of the playing interface when the player triggers the pitch playing controls. The audio waveform effects corresponding to different music styles differ in color, shape, or particle animation. For example, the audio waveform effect corresponding to the ancient style is a water wave diffusion animation with a cyan-green gradient, accompanied by falling petal particles, while the audio waveform effect corresponding to the electronic style is a digital spectrum columnar wave animation with a blue-purple gradient, accompanied by flashing square particles.

[0166] Based on the music style recognition results, the system selects a set of visual resources that matches the recognized music style from a set of preset visual resources. During the performance guidance process, the system loads the visual patterns of the pitch performance controls contained therein into the display attributes of each pitch performance control in the virtual instrument performance interface, replacing or overwriting the original default visual style, and configuring the audio waveform effects as feedback animations when the player triggers the pitch performance controls.

[0167] Following the aforementioned scenario where a player equips a virtual guqin in the town square and performs "free transcription," the player selects an audio file of a popular song as the transcription music information. During the performance guidance process, the system identifies the music style of the audio as "pop" using an audio classification model. The system then selects visual resources matching the "pop" style from a set of pre-set visual resources. In this example, the pitch performance control visual pattern corresponding to the pop style uses a simple solid color background with rounded corners, and the audio waveform effect uses a warm-colored gradient halo diffusion animation. The system loads these visual resources into the system during the performance guidance process. Figure 3In the virtual instrument performance interface shown, during the performance guidance process, the 21 pitch control buttons on the interface display visual patterns corresponding to popular music styles. Each time the player triggers a target pitch control button, a warm-colored halo spreading around the control appears as an audio waveform effect. If the same player previously selected a traditional Chinese style song for transcription, after the system recognizes the music style as "traditional Chinese style," it will load the corresponding wood grain background control pattern and ink ripple effect, making the visual presentation of the performance interface consistent with the atmosphere of traditional Chinese music.

[0168] As can be seen from the above explanation, the system identifies the musical style of the transcribed music information during the performance guidance process and loads matching visual resources based on the identification results, achieving automatic adaptation between the visual presentation of the performance interface and the style of the music content. This automated visual style matching mechanism allows the appearance and feedback effects of the performance interface to correspond with different styles of music, enhancing the player's visual immersion while following the performance guidance. It also eliminates the need for players to manually switch interface themes, improving the overall experience quality of the virtual instrument performance function.

[0169] Based on any embodiment of the method in this application, such as Figure 3 and Figure 11 As shown, in response to a touch operation command applied to the target pitch playing control, the pitch corresponding to the virtual instrument is played, including steps S1331 and S1322. Each step is explained below.

[0170] Step S1331: When guiding the performance using pitch performance sequence information, the player's actual touch events are compared with the pitch performance sequence information.

[0171] During the performance guidance process, in addition to providing visual cues to the performance controls for each target pitch based on the pitch performance sequence information, the system also monitors and compares the player's actual touch operations in real time. When a touch deviation exceeds a preset limit, the system automatically pauses or rewinds the performance progress, while displaying a recovery prompt to help the player re-enter the correct performance state. This fault-tolerant mechanism includes two continuously looping processing phases.

[0172] During the performance guidance, the system continuously tracks the current performance time and provides visual cues while simultaneously monitoring the player's touch interactions with various pitch control buttons on the virtual instrument performance interface. When a player triggers a pitch control on the virtual instrument performance interface through tapping, pressing, or using a controller button, the system captures the touch interaction and generates a corresponding actual touch event. This actual touch event includes the identifier information of the triggered pitch control and the actual timestamp of the touch.

[0173] The system compares each actual touch event with the corresponding target pitch performance control in the pitch performance sequence information to determine whether the actual touch event meets the preset performance requirements. The comparison process revolves around three dimensions. The first dimension is the control matching dimension, where the system determines whether the pitch performance control triggered in the actual touch event matches the target pitch performance control specified in the current performance time window in the pitch performance sequence information. If the two identifiers are the same, the actual touch event is judged as correct in the control matching dimension; if the two identifiers are different, it is judged as a touch error. The second dimension is the time matching dimension, where the system determines whether the actual occurrence timestamp of the actual touch event falls within the performance time window range of the corresponding target pitch performance control in the pitch performance sequence information. If the actual occurrence timestamp is between the start and end boundaries of the performance time window, the actual touch event is judged as correct in the time matching dimension; if the actual occurrence timestamp deviates from the performance time window, it is judged as a touch error. The third dimension is the omission detection dimension. When the end boundary of the performance time window of a target pitch performance control is reached, the system backtracks to check whether there is a corresponding actual touch event in that performance time window. If no actual touch event matching the target pitch performance control is detected during the performance time window, the system determines that a touch omission has occurred in that performance time window.

[0174] Reference Figure 3 The virtual instrument performance interface layout shown indicates that when the pitch performance sequence information specifies that a pitch performance control displaying the "Do" solfège syllable and shown at standard brightness is a triggerable control between 1000 and 1500 milliseconds, if the player clicks this control at 1200 milliseconds, the system determines that both the control matching and timing of the actual touch event are correct. If the player clicks a pitch performance control displaying the "Re" solfège syllable at 1200 milliseconds, the system determines that the actual touch event is a touch error. If, by the 1500-millisecond mark, the system has not detected any touch operation on the control displaying the "Do" solfège syllable, then it is determined that a touch omission has occurred during the performance time window.

[0175] In step S1332, if a touch error or omission is detected exceeding a predetermined threshold, the progression of pitch performance sequence information is automatically paused or rewound, and a recovery prompt mark is displayed.

[0176] While continuously comparing actual touch events with pitch performance sequence information, the system simultaneously accumulates statistics on the number of touch errors and missed touches. The preset threshold is a pre-defined numerical parameter used to define the minimum conditions for triggering the automatic recovery mechanism. The preset threshold can be set as the upper limit on the number of consecutive touch errors or missed touches, or as the upper limit on the cumulative total of touch errors and missed touches within a certain time period. For example, the preset threshold can be set to 3 consecutive missed touches, or 5 errors or missed touches accumulated within 10 seconds.

[0177] When the system detects that the cumulative statistics of touch errors or missed touches exceed a predetermined threshold, it automatically triggers a pause or rollback operation on the performance progress. A pause operation means the system immediately stops the progression of pitch performance sequence information, freezing the current performance time at the trigger moment, while maintaining the current visual prompts for each pitch performance control on the virtual instrument performance interface. A rollback operation means the system rewinds the current execution position of the pitch performance sequence information from the trigger moment to the last correctly performed node position, using the starting boundary of the performance time window of the target pitch performance control corresponding to that node position as the new starting point for recovery.

[0178] While a pause or rewind operation is being performed, the system displays a resume indicator on the virtual instrument playing interface. The resume indicator is a visual marker that indicates the starting point for resuming playing; it can appear as a flashing arrow, a highlighted box, or a prominent cursor, located on or near the target pitch playing control corresponding to the resume starting point. The visual style of the resume indicator is clearly distinct from the first and second visual indicators to avoid confusion with normal playing guidance. After observing the resume indicator, the player can readjust their finger position and continue playing from the target pitch playing control indicated by the indicator.

[0179] Reference Figure 3 The virtual instrument performance interface layout shown above, following the aforementioned business scenario, indicates that if a player misses three consecutive target pitch performance controls during performance, the system detects that the consecutive misses have exceeded a predetermined threshold of three and automatically pauses the progression of pitch performance sequence information. The system rewinds the current performance position to the end position corresponding to the last correctly triggered target pitch performance control before the consecutive misses, and displays a flashing orange arrow as a resumption prompt on the next target pitch performance control to be triggered immediately after that position. This flashing orange arrow clearly indicates to the player that the performance will restart from that target pitch performance control. Once the player is ready, clicking on the target pitch performance control or pressing the "Continue Playing" button on the interface will resume the progression of pitch performance sequence information, re-entering the normal performance guidance process.

[0180] As can be seen, during the performance guidance process, the system compares the actual touch events with the pitch performance sequence information in real time. When the touch deviation exceeds a preset limit, it automatically pauses or rewinds the performance progress and displays a recovery prompt, providing players with an intelligent performance error-tolerant mechanism. This mechanism avoids the accumulation of deviations and guidance misalignment caused by operational errors in traditional performance guidance methods. This allows players to quickly locate the recovery position and continue playing with the system's automatic assistance, without manually interrupting the performance and starting from the beginning after a performance error. This improves the error tolerance of the virtual instrument performance function and the smoothness of the performance experience.

[0181] Based on any embodiment of the method in this application, such as Figure 3 and Figure 12 As shown, the response to the touch operation command applied to the target pitch playing control includes steps S1341 and S1342, and each step is explained below.

[0182] Step S1341: Collect touch time deviation data generated by multiple players when playing according to the pitch order information.

[0183] After the player completes the performance guidance and plays the corresponding pitch of the virtual instrument, the system also performs a background feedback optimization process. This process collects touch timing deviation data generated by multiple players during actual performances and adjusts the transcription constraints in the prompt word template based on the aggregated deviation data to change the transcription accuracy or beat quantization granularity of the subsequently generated structured score.

[0184] Touch time deviation data refers to the time difference between the actual moment a player triggers the target pitch performance control during the performance guidance and the reference time of the performance time window specified in the pitch performance sequence information. As the system continuously compares the player's actual touch events with the pitch performance sequence information during the performance guidance, for each correctly determined touch event, the system simultaneously calculates the offset between the timestamp of the actual touch event and the start time of the corresponding performance time window. This offset is the touch time deviation data. Touch time deviation data can contain positive and negative signs; a positive number indicates that the player's actual touch time is later than the reference time, and a negative number indicates that the player's actual touch time is earlier than the reference time.

[0185] The system generates a set of touch timing deviation data for each complete performance. This set contains the timing deviation values ​​corresponding to all correct touch events during the performance, as well as the pitch and rhythm type information associated with each deviation value. For example, a deviation value can be associated with a pitch control with a specific solfège syllable, or with a specific rhythmic pattern, such as continuous eighth notes or dotted rhythms. This additional information provides multi-dimensional data support for subsequent aggregate analysis.

[0186] The collection of touch timing deviation data can cover multiple performance records from multiple player groups. In practice, after each performance, the player's terminal device uploads the resulting touch timing deviation data set to the game service cluster via the network. The game service cluster persistently stores the massive amounts of touch timing deviation data from different players, different terminal devices, and different time periods. The system can also preprocess the collected touch timing deviation data to remove obvious abnormal deviation values ​​caused by accidental operation errors or device response delays, ensuring the data quality for subsequent aggregation analysis.

[0187] Step S1342: Based on the aggregated touch time deviation data, adjust the time quantization constraints or target pitch range constraints in the prompt word template to change the transcription accuracy or beat quantization granularity of the subsequently generated structured musical score.

[0188] The system aggregates and analyzes touch timing deviation data collected from multiple players to identify common operational deviation patterns. This aggregation analysis can be conducted along different dimensions. In terms of time offset, the system statistically analyzes the mean and variance distributions of all touch timing deviation data to determine if the player group as a whole exhibits a tendency to prematurely or delayedly trigger touches. In terms of pitch, the system groups and statistically analyzes deviation data by pitch control to identify significant concentrations of deviation at specific pitches or solfège syllables. In terms of rhythm, the system groups and statistically analyzes deviation data by rhythm type to identify common deviation patterns within specific rhythmic patterns. For example, the aggregation analysis might reveal that most players commonly experience delayed triggering when playing consecutive sixteenth note passages, or exhibit significant deviations when playing specific solfège syllables in the high register.

[0189] Based on the results of the aggregation analysis, the system adjusts the transcriptional constraints in the prompt word template. The adjustments target either the time quantization constraints or the target pitch range constraints in the prompt word template. When the aggregation analysis shows a widespread triggering deviation among players in specific rhythmic patterns, the system adjusts the time quantization constraints. For example, if most players exhibit significant time deviations in sixteenth-note passages, the system adjusts the time quantization constraints from a sixteenth-note grid to an eighth-note grid, resulting in a more granular quantization of such rhythmic patterns in the subsequently generated structured score, making it easier for most players to follow along. When the aggregation analysis shows a relatively concentrated triggering deviation among players in a specific pitch range, the system adjusts the target pitch range constraints. For example, if most players show significant deviations when triggering pitch control in the high register, the system appropriately lowers the upper limit of the target pitch range constraints, resulting in more notes in the subsequently generated structured score being distributed within the middle register where players can operate more accurately.

[0190] Adjustments to transcription constraints alter the transcription precision or beat quantization granularity of the subsequently generated structured music scores. Changes in transcription precision are reflected in the accuracy of note timing within the structured score, while changes in beat quantization granularity are reflected in the fineness of the beat grid followed by note start times. The adjusted transcription constraints are written into the corresponding slots of the prompt word template. When a player initiates another music transcription request and generates new pitch performance order information, the new pitch performance order is generated based on the adjusted transcription constraints, and its note timing arrangement is closer to the actual playing habits of most players.

[0191] Building upon the aforementioned business scenarios, the system collected data from multiple players. Figure 3 The data shows the touch timing deviation when playing a virtual musical instrument on the interface following the pitch order information generated by "free transcription". Aggregate analysis results show that when playing consecutive sixteenth note passages, most players' actual trigger times were delayed by an average of approximately 50 to 80 milliseconds compared to the baseline time. Based on this, the system adjusted the time quantization constraints in the prompt template, changing them from a sixteenth note grid to an eighth note grid. When one player uses the "free transcription" function again to transcribe a new piece of music, the system uses the adjusted prompt template to generate the score prompt text. In the structured score generated by the large language model under the constraints, the start times of the notes are all quantized to the eighth note grid, eliminating dense rhythms at the sixteenth note level. The generated pitch order information is more consistent with the actual operating habits of most players.

[0192] As can be seen, the system collects touch timing deviation data from multiple players and adjusts the transcriptional constraints in the prompt word template based on the aggregated analysis results, achieving adaptive feedback optimization from group performance behavior data to score generation strategy. This data-driven optimization mechanism makes the subsequently generated pitch performance order information closer to the actual operating habits of most players in terms of temporal accuracy and pitch range distribution, reducing performance deviations caused by individual differences and improving the group adaptability and overall experience quality of the virtual instrument performance guidance function.

[0193] Please see Figure 13According to one aspect of this application, a virtual musical instrument playing device includes: a music transcription request module 11, a playing sequence generation module 12, and a playing guidance processing module 13. The music transcription request module 11 is used to, in response to a music transcription request, acquire and transcribe music information to generate a structured score, which includes pitch information and pitch timing information. The playing sequence generation module 12 is used to determine the target pitch playing control corresponding to the pitch information in the virtual musical instrument, and based on the pitch timing information, determine the playing time window of the target pitch playing control to generate corresponding pitch playing sequence information. The playing guidance processing module 13 is used to, in response to a playing instruction, provide playing guidance based on the pitch playing sequence information, and during the playing guidance process, respond to touch operation instructions acting on the target pitch playing control and play the pitch corresponding to the touch operation instruction.

[0194] Based on any embodiment of the device in this application, the music transcription request module 11 includes: a cappella recording unit, used to perform chord analysis on the player's cappella audio and extract chord progression information; a chord analysis unit, used to generate at least one accompaniment track based on the chord progression information and the rhythmic characteristics of the player's cappella audio, the accompaniment track containing a sequence of accompaniment notes synchronized with the beat of the player's cappella audio; and an audio synthesis unit, used to synthesize at least one accompaniment track into accompaniment audio, so as to synchronously play the accompaniment audio and the player's cappella audio during the performance guidance.

[0195] Based on any embodiment of the device in this application, the music transcription request module 11 includes: a constraint rule storage unit for storing combinations of transcription constraint conditions corresponding to different performance difficulties and musical style preferences, wherein the performance configuration parameters include at least one of performance difficulty and musical style preference, and the transcription constraint conditions include at least one of output format constraint, time quantization constraint, target pitch range constraint and voice part restriction constraint; and a constraint condition search unit for searching for the corresponding combination of transcription constraint conditions from the constraint rule storage unit according to the obtained performance configuration parameters.

[0196] Based on any embodiment of the device in this application, the music transcription request module 11 includes: a sound source separation unit, used to perform sound source separation on the transcribed music audio before inputting the score prompt text and the transcribed music audio into the large language model, to obtain at least two independent voice tracks; and a voice selection unit, used to receive the player's selection instruction for the independent voice tracks, so as to automatically transcribe the selected voice.

[0197] Based on any embodiment of the device in this application, the music transcription request module 11 includes: an overlimit chord detection unit, used to detect whether there are overlimit chords in the structured score, where an overlimit chord is a chord in the structured score whose number of pitches appearing simultaneously exceeds the upper limit of the number of operation controls that can be triggered simultaneously by the virtual instrument; and a chord simplification unit, used to convert the overlimit chord into a playable single-note sequence or arpeggio sequence according to a preset simplification strategy when an overlimit chord exists; wherein, a single-note sequence refers to a sequence generated by retaining the highest note or root note of the overlimit chord, and an arpeggio sequence refers to a sequence generated by decomposing the overlimit chord into arpeggios according to a preset musical order.

[0198] Based on any embodiment of the device in this application, the music transcription request module 11 further includes: a music structure analysis unit, used to perform music structure analysis on the transcribed music information before performance guidance based on pitch performance order information, identify multiple performance segments and generate corresponding segment tags; and a segment tag display unit, used to display the segment tag of the current performance segment when performance guidance is performed based on pitch performance order information, wherein the segment tag is used to characterize the music segment in the transcribed music information that corresponds to the current performance progress.

[0199] Based on any embodiment of the device in this application, the music transcription request module 11 further includes: a multi-route display unit for displaying an operation route switching control, wherein the operation route switching control is used to display at least two operation routes for the player to select and enable; and a route switching control providing unit for responding to the route selection instruction of the operation route switching control, taking the selected operation route corresponding to the route selection instruction as the final operation route, and providing performance guidance based on the pitch performance order information corresponding to the final operation route.

[0200] Based on any embodiment of the device in this application, the performance guidance processing module 13 includes: a first visual prompt display unit, used to determine the first target pitch performance control corresponding to the performance time window into which the current time falls, and to display a first visual prompt effect on the first target pitch performance control within the performance time window; a second visual prompt display unit, used to display a second visual prompt effect on the second target pitch performance control corresponding to the next performance time window with a visual intensity weaker than the first visual prompt effect; and a visual intensity transition unit, used to gradually increase the visual intensity of the second visual prompt effect inversely proportional to the time difference when the time difference between the current time and the start time of the performance time window of the second target pitch performance control is less than a preset threshold, until the time difference is zero, and then switch to the first visual prompt effect.

[0201] Based on any embodiment of the device in this application, the performance guidance processing module 13 further includes: a music style recognition unit, used to identify the music style of the transcribed music audio; a visual resource matching unit, used to select visual resources that match the music style from a set of preset visual resources and load them into the virtual instrument; wherein, the visual resources include at least one of the following: visual patterns of operation controls, the visual patterns corresponding to different music styles are different in texture, color or border style; audio waveform effects presented when the user triggers the operation controls, the audio waveform effects corresponding to different music styles are different in color, shape or particle animation.

[0202] Based on any embodiment of the device in this application, the performance guidance processing module 13 further includes: a touch event comparison unit, used to compare the player's actual touch events with the pitch performance order information when providing performance guidance using pitch performance order information; and a performance recovery unit, used to automatically pause or rewind the progression of pitch performance order information and display a recovery prompt mark if a touch error or omission is detected exceeding a predetermined threshold.

[0203] Based on any embodiment of the device in this application, the performance guidance processing module 13 further includes: a deviation data collection unit, used to collect touch time deviation data generated by multiple players when performing according to pitch performance order information; and a prompt word template adjustment unit, used to adjust the transcription constraints in the prompt word template based on the aggregated touch time deviation data, so as to change the transcription accuracy or beat quantization granularity of the subsequently generated structured musical score. Another embodiment of this application also provides a virtual musical instrument performance device.

[0204] like Figure 14 The diagram shows the internal structure of a virtual musical instrument playing device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database stores information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement a virtual musical instrument playing method.

[0205] The processor of this virtual musical instrument playing device provides computing and control capabilities to support the operation of the entire device. The device's memory can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the virtual musical instrument playing method of this application. The network interface of the virtual musical instrument playing device is used for communication with a terminal.

[0206] Those skilled in the art will understand that Figure 14The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the virtual musical instrument playing device to which the present application is applied. A specific virtual musical instrument playing device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0207] In this embodiment, the processor is used to execute... Figure 13 The specific functions of each module are described, and the memory stores the program code and various data required to execute the aforementioned modules or sub-modules. A network interface is used to enable data transmission between user terminals or the server. In this embodiment, the non-volatile readable storage medium stores the program code and data required to execute all modules in the virtual musical instrument playing device of this application. The server can call the server's program code and data to execute the functions of all modules.

[0208] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the virtual musical instrument playing method of any embodiment of this application.

[0209] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the methods in any embodiment of this application.

[0210] In summary, this application provides players with a guide to converting audio into virtual musical instruments, lowering the barrier to entry for players and enriching the sources of playable content.

[0211] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

Claims

1. A method for playing a virtual musical instrument, characterized in that, include: In response to a music transcription request, the music transcription information is obtained and the music is transcribed to generate a structured musical score, which includes pitch information and pitch timing information. The target pitch performance control corresponding to the pitch information in the virtual musical instrument is determined, and the performance time window of the target pitch performance control is determined based on the pitch timing information, and the corresponding pitch performance sequence information is generated. In response to a performance instruction, performance guidance is provided based on the pitch performance sequence information, and during the performance guidance process, the pitch corresponding to the touch operation instruction is played in response to a touch operation instruction applied to the target pitch performance control.

2. The method according to claim 1, characterized in that, The transcribed music information includes a cappella audio of the player, and the method further includes: Perform chord analysis on the player's a cappella audio and extract chord progression information; Based on the chord progression information and the rhythmic characteristics of the player's a cappella audio, at least one accompaniment track is generated, the accompaniment track containing a sequence of accompaniment notes synchronized with the beat of the player's a cappella audio; The at least one accompaniment track is combined into an accompaniment audio, which is then played synchronously with the player's a cappella audio during the performance guidance.

3. The method according to claim 1, characterized in that, The method further includes: Before the performance guidance based on the pitch performance sequence information, the transcribed music information is analyzed for music structure to identify multiple performance segments and generate corresponding segment tags; When guiding the performance based on the pitch performance order information, the segment label of the current performance segment is displayed. The segment label is used to characterize the music segment in the transcribed music information that corresponds to the current performance progress.

4. The method according to claim 1, characterized in that, The pitch performance sequence information has multiple values, and each pitch performance sequence information corresponds to an operation route. The method further includes: The operation route switching control is used to display at least two operation routes for the player to select and enable. In response to the route selection command of the operation route switching control, the selected operation route corresponding to the route selection command is taken as the final operation route, and the performance guidance is performed based on the pitch performance sequence information corresponding to the final operation route.

5. The method according to claim 1, characterized in that, The performance guidance based on the pitch performance sequence information includes: Determine the first target pitch performance control corresponding to the performance time window in which the current time falls, and make the first target pitch performance control display a first visual cue effect within the performance time window; A second visual cue effect is displayed on the second target pitch performance control corresponding to the next performance time window, and the visual intensity of the second visual cue effect is weaker than that of the first visual cue effect. When the time difference between the current time and the start time of the performance time window of the second target pitch performance control is less than a preset threshold, the visual intensity of the second visual cue effect is gradually increased inversely to the time difference until the time difference is zero, at which point the first visual cue effect is switched.

6. The method according to claim 1, characterized in that, The performance guidance based on the pitch performance sequence information also includes: Identify the music style corresponding to the transcribed music audio information; Select visual resources that match the music style from a set of preset visual resources, and load and display the visual resources during the performance guidance process.

7. The method according to claim 1, characterized in that, The method further includes responding to touch operation commands applied to the target pitch performance control. When guiding the performance using the pitch performance order information, the player's actual touch events are compared with the pitch performance order information. If a touch error or omission is detected exceeding a predetermined threshold, the progression of the pitch performance sequence information will be automatically paused or reverted, and a recovery prompt will be displayed.

8. A virtual musical instrument playing device, characterized in that, include: The music transcription request module is used to respond to a music transcription request, obtain music transcription information, perform music transcription, and generate a structured musical score, wherein the structured musical score includes pitch information and pitch timing information; The performance sequence generation module is used to determine the target pitch performance control corresponding to the pitch information in the virtual instrument, and based on the pitch timing information, determine the performance time window of the target pitch performance control and generate the corresponding pitch performance sequence information. The performance guidance processing module is used to respond to performance instructions, perform performance guidance based on the pitch performance sequence information, and, during the performance guidance process, respond to touch operation instructions applied to the target pitch performance control and play the pitch corresponding to the touch operation instruction.

9. A virtual musical instrument playing device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images