Music game method, device and system, electronic equipment and storage medium

Abstract

The invention provides a music game method, device and system, electronic equipment and a storage medium. Game configuration information (including target range space configuration information, target energy body configuration information and target associated musical instrument configuration information) of a current music track is determined, and interaction is performed based on notes, hand positions and finger actions; according to the method, unified modeling and synchronous visualization of the music multi-dimensional parameters are realized, the problems of single dimension, audio-visual disjunction and low interaction efficiency in the prior art are effectively solved, unified modeling and synchronous visualization can be performed on the music multi-dimensional parameters, accurate space interaction is supported, and immersion of user experience and music education value are effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of music game technology, and more particularly to a music game method, apparatus, system, electronic device, and storage medium. Background Technology

[0002] Against the backdrop of the rapid development of extended reality (XR) technology, existing music applications suffer from a significant disconnect between the visualization and interaction of music parameters. Currently, mainstream XR music applications primarily take two forms: The first type is XR instrument simulation applications (such as virtual pianos and guitars). While these applications achieve high-precision 3D reconstruction of the instrument's shape, their interaction logic essentially involves directly transferring the interface of the physical instrument to the virtual space. User interaction remains limited to triggering notes on fixed "keys" or "strings," failing to transform deep musical parameters such as pitch, range, and chord progressions into spatial structures and dynamic visual feedback that users can freely explore. Therefore, the user experience is more like "playing a physical instrument in a virtual world," and the spatial perception and immersive rendering advantages of XR technology are not used to deconstruct and express the intrinsic multidimensional information of music.

[0003] The second type is XR music rhythm games (such as rhythm-based hitting or lightsaber games). The core interaction mechanism of this type of application is "rhythm hit," where players interact with visual objects that fly along a fixed trajectory at specific times. Their visual cues (such as the timing of the object's appearance or its speed) are usually strongly correlated only with the temporal rhythm dimension of the music, and completely decoupled from other musical parameters such as pitch, timbre, and range. Even with the music turned off, the game can still be played as a pure rhythm-based game. This results in an extremely limited visualization dimension, failing to convey any musical information other than rhythm through visuals. This disconnects the interaction process from the artistic connotations of the music, greatly limiting its potential application in music education.

[0004] Both of these mainstream forms expose a core flaw in existing technology: the lack of a unified model for multi-dimensional musical parameters (pitch, range, scale, rhythm, instrument timbre, etc.) and a mechanism for synchronously mapping them to the XR visual space for dynamic rendering. Consequently, the system cannot achieve unified audiovisual feedback across these parameters, resulting in a weak connection between user interaction and the deep structure of the music. This leads to a simplistic and inefficient human-computer interaction path, ultimately severely limiting the potential of XR environments to provide deep immersion and music education value. Summary of the Invention

[0005] In view of this, in order to solve at least some of the above-mentioned technical problems, this disclosure provides a music game method, apparatus, system, electronic device and storage medium.

[0006] According to a first aspect of the present disclosure, a music game method is provided, the music game method comprising: Determine the game configuration information for the current music track; wherein, the game configuration information includes at least the target pitch range configuration information, target energy body configuration information, and target associated instrument configuration information corresponding to the current music track; Based on the game configuration information and the current note of the current music track, the target pitch space and target energy body corresponding to the current note are determined, as well as the target associated instrument corresponding to the segment in which the current note is located in the target pitch space; wherein, the target pitch space is the pitch space corresponding to the current note, and the target energy body is the energy body corresponding to the current note; After determining that the player's hand position is within the target sound range space of the player's XR visual space, the display state of the target energy body is controlled by the visual model corresponding to the target associated musical instrument, and the target energy body moves in the target sound range space toward the direction of the set judgment line; When the target energy body moves to the set judgment line, the game result of the current note is determined based on the time of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line.

[0007] Optionally, the game scene in the XR visual space includes multiple sound domains, each sound domain corresponds to a multiple pitch range, and each sound domain corresponds to at least one associated instrument. The multiple sound domains are configured with different scene images. The game configuration information for determining the current music track includes: Based on the pitch-range space mapping table and the pitches of all notes in the current music piece, the target range space configuration information corresponding to the current music piece is determined; wherein, the pitch-range space mapping table includes the mapping relationship between multiple pitch ranges and multiple range spaces; Based on the pitch space-associated instrument mapping table and the target pitch space configuration information, the target associated instrument information corresponding to the target pitch space configuration information is determined.

[0008] Optionally, each of the sound domain spaces includes seven motion tracks corresponding to the seven energy bodies, the seven energy bodies correspond to the seven musical scales, and the seven energy bodies are each configured with a different color; The game configuration information for determining the current music track includes: Based on the scale-energy mapping table and all notes in the current music piece, the target energy information corresponding to the current music piece is determined.

[0009] Optionally, determining the game result of the current note based on the moment of the player's most recent set gesture or any finger press action, and the target moment associated with the target energy body closest to the set judgment line, includes: Determine the time difference ΔT between the occurrence time and the target time; If |ΔT| ≤ 50ms, then the game result of the current note is determined to be a perfect hit, the highest score is recorded, and the first special effect is triggered; If 50ms < |ΔT| ≤ 100ms, then the game result of the current note is determined to be a hit, the base score is recorded, and the second effect is triggered; If |ΔT|>100ms, then the game result of the current note is determined to be a mistake and the combo is interrupted, and no points are scored.

[0010] Optionally, The music game method includes: When the newly determined target sound domain space is different from the current target sound domain space, a prompt message to switch sound domain space is displayed in the XR visual space during the first time period before the target energy body reaches the set judgment line. When the newly determined target pitch space differs from the current target pitch space, the game result for the current note is determined based on the occurrence time of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line, including: After outputting the prompt information, if it is detected that the player's hand position has not been adjusted, and the player's hand position at the time of occurrence is outside the target pitch range space, then the game result of the current note is determined to be a game failure; After the prompt message is output, if it is detected that the player's hand position has been adjusted, and the player's hand position at the time of the occurrence is outside the target pitch range, then the game result of the current note is determined to be a mistake and the combo is interrupted, and no points are scored.

[0011] Optionally, the music game method includes: During the game, if the set conditions are met, the AI ​​improvisation game mode will be triggered. In the AI ​​improvisation game mode, by setting an AI model, the music segments above and below the current note of the current music piece, the chord progression of the current music piece, the music type, the target pitch range space corresponding to the current note, and the target associated instrument are processed to obtain an improvisational music segment. Pause the game process for the current music track and start the game process for the improvised music segment.

[0012] According to a second aspect of the present disclosure, a music game device is provided, the music game device comprising: The determination module is used to determine the game configuration information of the current music track; wherein, the game configuration information includes at least the target pitch range space configuration information, target energy body configuration information, and target associated instrument configuration information corresponding to the current music track; The determining module is further configured to determine, based on the game configuration information and the current note of the current music track, the target pitch space and target energy body corresponding to the current note, and the target associated instrument corresponding to the segment in which the current note is located in the target pitch space; wherein, the target pitch space is the pitch space corresponding to the current note, and the target energy body is the energy body corresponding to the current note; The rendering module is used to determine that the player's hand position is within the target sound range space of the player's XR visual space, and then control the display state of the target energy body with the visual model corresponding to the target associated musical instrument, and the target energy body moves in the target sound range space toward the direction of the set judgment line; The determining module is further configured to determine the game result of the current note based on the time of occurrence of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line when the target energy body moves to the set judgment line.

[0013] According to a third aspect of the present disclosure, a music game system is provided, the music game system including XR glasses and an electromyography (EMG) bracelet, the music game system being used to implement the music game method as described in any of the first aspects.

[0014] According to a fourth aspect of the present disclosure, an electronic device is provided, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the music game method as described in any of the first aspects.

[0015] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the music game method as described in any of the first aspects.

[0016] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: In this disclosure, by determining the game configuration information of the current music track (including target pitch space configuration information, target energy body configuration information, and target associated instrument configuration information) and interacting based on notes, hand positions, and pressing actions, unified modeling and synchronous visualization of multi-dimensional music parameters are realized. This effectively solves the problems of single dimension, audiovisual disconnect, and low interaction efficiency in the prior art. It has the ability to uniformly model and synchronously visualize multi-dimensional music parameters, support precise spatial interaction, and effectively enhance the immersive experience and music education value of the user experience.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a flowchart illustrating a music game method according to an exemplary embodiment.

[0020] Figure 2 This is a block diagram of a music game device according to an exemplary embodiment.

[0021] Figure 3 This is a block diagram of an electronic device according to an exemplary embodiment.

[0022] Figure 4 This is a schematic diagram of the game scene distribution of a music game method according to an exemplary embodiment. Detailed Implementation

[0023] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0024] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0025] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0026] In traditional XR music applications, the problem of limited visualization dimensions for multi-dimensional music parameters manifests as the system only being able to process rhythm or a single pitch dimension. This results in the inability to uniformly model and synchronously present multi-dimensional parameters such as pitch, range, scale, and rhythm. The essence of this problem lies in the lack of a structured encapsulation mechanism for the comprehensive information of music in existing technologies. This prevents user interactions from establishing spatial relationships with multi-dimensional parameters, leading to redundant and inefficient human-computer interaction paths. Furthermore, the lack of low-latency collaborative rendering of multi-dimensional parameters at the system level causes a disconnect between audiovisual feedback and temporal and spatial dimensions, directly impacting the ability to maintain immersion in the XR environment and the depth of music education functionality.

[0027] If the above problems are not resolved, XR music applications will be unable to support the multi-dimensional parameter analysis of complex musical works, resulting in continuous information transmission gaps when the system processes polyphonic or transposed pieces; low interaction efficiency will further weaken the correlation between user operations and music parameters, hindering the application of spatial navigation capabilities in music learning; the disconnect between audiovisual feedback will persist for a long time, continuously weakening the foundation for building immersion in the XR environment, thereby limiting the expansion potential of this technology in professional music education and high-fidelity entertainment scenarios.

[0028] In this regard, refer to Figure 1 and Figure 4 As shown, this application proposes a music game method including: S110. Determine the game configuration information for the current music track; wherein, the game configuration information includes at least the target pitch range configuration information, target energy body configuration information, and target associated instrument configuration information corresponding to the current music track; S120. Based on the game configuration information and the current note of the current music track, determine the target pitch space and target energy body corresponding to the current note, as well as the target associated instrument corresponding to the segment in which the current note is located in the target pitch space; wherein, the target pitch space is the pitch space corresponding to the current note, and the target energy body is the energy body corresponding to the current note; S130. After determining that the player's hand position is within the target sound range space of the player's XR visual space, the display state of the target energy body is controlled by the visual model corresponding to the target associated musical instrument, and the target energy body moves in the target sound range space toward the direction of the set judgment line. S140. When the target energy body moves to the set judgment line, the game result of the current note is determined based on the time of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line.

[0029] Among them, game configuration information can be understood as a set of visual parameters that define the current music track in the extended reality visual space. It can be configured through predefined rules or dynamic analysis. For example, the range boundary value can be manually input based on the user interface to divide the range space, or the energy body parameters can be determined through audio feature extraction algorithms. Its main purpose is to achieve a systematic encapsulation of multi-dimensional music parameters such as pitch, range, scale and instrument.

[0030] Specifically, the target pitch range configuration information can be configured to use a user-defined pitch range setting method, such as specifying the low pitch range as C2 to B3 and the middle pitch range as C4 to B4; the target energy body configuration information can be implemented using a randomly generated or fixed-configuration color coding scheme, such as assigning random colors to different scales; the target associated instrument configuration information refers to the configuration information of associated instruments corresponding to the target pitch range configuration information, for example, each pitch range space is configured with at least one associated instrument.

[0031] Specifically, determining the target pitch range based on game configuration information and the current note can be implemented by comparing the note pitch with the configured pitch range, for example, by matching the pitch to a preset pitch range using a lookup table; or by determining the pitch range to which the note belongs through real-time frequency calculation. Detecting the player's hand position can be implemented using the visual sensors of an extended reality device, for example, by using camera images and computer vision algorithms to estimate hand posture; or by acquiring hand motion data through an external inertial measurement unit.

[0032] When the player's hand is positioned within the target sound range space, the display state of the target energy body can be controlled by the visual model corresponding to the target-associated musical instrument. This can be achieved by using 3D graphics rendering technology, such as loading a preset 3D musical instrument model to indicate the energy body's behavior; or by using a particle system to simulate the musical instrument playing effect.

[0033] The XR visual space can be realized using XR devices such as smart glasses, which typically serve as the display end. The detection of the player's set gestures or arbitrary finger presses can be achieved through the input end. For example, set gestures or arbitrary finger presses can be detected using a handheld physical controller, which provides multiple buttons and vibration feedback mechanisms. However, considering the inconvenience of carrying a handheld physical controller, the detection of set gestures or arbitrary finger presses can also be done using a bare-hand approach, such as gesture capture. However, considering the potential delays and occlusion issues with gesture capture, finger movements can also be detected using an electromyography (EMG) wristband. This can be implemented by using a surface electromyography (SEMG) sensor to collect muscle electrical signals and identifying presses based on signal thresholds; or by classifying gesture types using pattern recognition algorithms. EMG wristbands are portable, have no delay, do not have occlusion issues, provide feedback, and can be set with multiple fingers. It should be noted that in this embodiment, the display end and input end are integrated into a single XR product to implement the entire music game method.

[0034] This embodiment solves the problems of single visualization dimension, low interaction efficiency, and disconnect between audiovisual feedback in existing technologies by integrating unified modeling and spatial interaction mechanisms for multi-dimensional music parameters. Specifically, the determination of game configuration information avoids the limitations of traditional solutions that only focus on rhythm or a single pitch, providing a basic framework for the transmission of multi-dimensional information; the dynamic determination of visualization elements based on configuration information and the current note ensures that the generation of the sound domain space and energy body is closely dependent on the overall structure of the music piece, thereby achieving synchronous rendering of multi-dimensional parameters; the binding of the player's hand position to the sound domain space allows users to intuitively understand changes in the sound domain through natural spatial navigation; the low latency of electromyography ensures the accurate capture of the moment of any finger pressing action, ensuring the real-time correlation between interactive feedback and music parameters, thereby enhancing immersion.

[0035] The working principle of this music game method involves the systematic processing and spatial interaction of multi-dimensional musical parameters. First, the game configuration information for the current music track is determined, including at least the target pitch range configuration information, target energy body configuration information, and target associated instrument configuration information. This determination enables unified modeling of multi-dimensional musical parameters such as pitch, range, scale, and instruments, avoiding the limitations of traditional techniques that only focus on rhythm or a single pitch, thus building a basic framework for the transmission of multi-dimensional information. Further, based on the game configuration information and the current note of the current music track, the target pitch range space and target energy body corresponding to the current note are determined, as are the target associated instruments corresponding to the segment in the target pitch range space. This process dynamically maps notes to specific visual elements through the contextual relevance of the configuration information, ensuring that the generation of the pitch range space and energy body is closely dependent on the overall structure of the music track, rather than processing individual parameters in isolation. This solves the problem of audiovisual feedback disconnect caused by the lack of synchronous rendering of multi-dimensional parameters. When the player's hand is positioned within the target pitch range of the XR visual space, the display state of the target energy body is controlled by the visual model corresponding to the instrument associated with the target, and the target energy body moves towards the set judgment line within the target pitch range. This feature, by binding the hand's spatial position to the pitch range area and combining it with the instrument's specific visual model, allows users to intuitively understand pitch range changes through natural hand navigation, significantly improving the intuitiveness and efficiency of the interaction. When the target energy body moves to the set judgment line, the game result of the current note is determined based on the timing of the set gesture or any finger pressing action and the target time associated with the target energy body. The low-latency characteristics of electromyography are used to accurately capture the user's operation time point, and rhythm determination is achieved through time difference calculation, ensuring real-time synchronization between interactive feedback and music parameters.

[0036] It should be noted that, in this embodiment, "segment" refers to a musical segment divided based on musical form (e.g., phrase, section) or a preset time window (e.g., 4 or 8 measures). The system can assign at least one associated instrument to each segment based on music analysis or preset tags.

[0037] In a specific embodiment, the visual model corresponding to the target associated musical instrument can be a three-dimensional model of a harp (denoted as the harp model), and the target energy body can be a luminous harp model. For example, when playing a classical music piece, the system determines the game configuration information of the piece, where the target pitch space configuration information defines three spaces: low pitch, mid pitch, and high pitch; the target energy body configuration information includes various energy bodies; and the target associated musical instrument configuration information specifies the harp as the associated musical instrument. For the current note A4, the system determines its target pitch space to be the high pitch space, and the target energy body to be the luminous harp model corresponding to A4. When the player's hand enters the high pitch space area of ​​the XR visual space, the energy body begins to move towards the set judgment line. When the energy body moves to the set judgment line, the time difference between the time of the set gesture or any finger pressing action and the target time is calculated to determine the game result of the current note.

[0038] This embodiment transforms abstract musical parameters into perceptible spatial entities, breaking the limitations of single-dimensional expression and enhancing the immersive experience of audiovisual interaction. It allows users to intuitively perceive the deep structure of music, and the display state of corresponding energy bodies is controlled by the visual model of the associated musical instrument. The display of energy bodies is bound to the timing and timbre of notes. The game result is determined by combining the timing of a set gesture or any finger press with the target note, forming a three-dimensional interactive logic of "spatial position + visual form + temporal rhythm." This strongly couples interactive actions with multi-dimensional musical parameters, enhancing the musical relevance of the interaction. In this embodiment, users can establish pitch range cognition by exploring the pitch space, distinguish timbre by recognizing energy body shapes, and understand note timing by accurately hitting the judgment line, achieving "interaction as learning." This breaks the limitations of traditional applications that prioritize entertainment over cognition, providing core support for XR music enlightenment and music theory teaching. Furthermore, it can better adapt to diverse scenario needs, supporting customization of pitch space, energy bodies, and judgment rules according to song style and user groups, combining flexibility and universality, covering multiple scenarios such as entertainment and education; at the same time, it simplifies the interaction path, reduces learning costs, and improves human-computer interaction efficiency and user experience smoothness.

[0039] In other words, this technical solution achieves complete visualization of multi-dimensional music parameters through the aforementioned mechanism, overcoming the shortcomings of existing technologies that only offer a single visualization dimension. It allows for the simultaneous presentation of multi-dimensional information such as pitch, range, scale, and instruments. Users can intuitively understand and control changes in the range through natural hand navigation in the XR visual space, effectively improving interaction efficiency and solving the problem of low human-computer interaction efficiency. Simultaneously, the low-latency detection of the myoelectric bracelet ensures accurate capture of the moment finger pressure occurs, guaranteeing real-time synchronization between interactive feedback and music parameters. This eliminates the disconnect between audiovisual feedback and visual experience, thereby enhancing the user's immersive experience.

[0040] In some of the embodiments described above in this application, game configuration information is proposed to define parameters such as the target pitch space. However, in its implementation, there is a lack of a systematic mapping mechanism to dynamically map the pitch range of the music track to the pitch space and associate it with the instrument. This results in the inability to achieve spatial visualization of the pitch dimension, causing incomplete transmission of multi-dimensional music parameters and a disconnect between interactive actions and pitch changes, thereby weakening the sense of immersion and educational depth.

[0041] In this regard, this application further proposes the following steps: The game scene in the XR visual space includes multiple sound domains, each corresponding to a different pitch range, and each sound domain is associated with at least one related instrument. Different scene images are configured for each of the multiple sound domains. In this embodiment, determining the game configuration information for the current music track may include: S210. Based on the pitch-range space mapping table and the pitches of all notes in the current music piece, determine the target range space configuration information corresponding to the current music piece; wherein, the pitch-range space mapping table includes the mapping relationship between multiple pitch ranges and multiple range spaces. S220, Based on the pitch space-associated instrument mapping table and the target pitch space configuration information, the target associated instrument information corresponding to the target pitch space configuration information.

[0042] In practical applications, the pitch space refers to the discretized operational area that maps the range of musical pitches into the XR visual space. It can be implemented by dividing the pitch range based on continuous intervals or by grouping based on scale clustering. The purpose is to transform continuous pitch signals into visual operational units with clear boundaries, so that players can directly perceive pitch changes by moving their hands in space.

[0043] The pitch-range space mapping table can be understood as a data structure that defines the rules for converting pitch ranges to visual space regions. Specifically, it can employ linear proportional mapping or non-linear segmented threshold mapping. Its purpose is to resolve the ambiguity issues in the discretization of continuous pitches and ensure the consistency of range division across different pieces. Furthermore, the range space-associated instrument mapping table can be specifically an index table storing the correspondence between range space identifiers and instrument model identifiers. Its purpose is to dynamically bind visual feedback to musical elements, avoiding a disconnect between audiovisual feedback and enhancing the educational expression of musical parameters.

[0044] Specifically, the proposed solution uses a pitch-range space mapping table to batch convert the pitches of all notes in the current music track into target range space configuration information. This mapping table defines a standardized correspondence between pitch ranges and range spaces, enabling pitch changes to form identifiable boundary areas in visual space. Subsequently, the system dynamically associates the determined range space configuration information with the corresponding associated instruments based on the range space-associated instrument mapping table, ensuring that the energy body triggers the corresponding instrument visual model during movement, so that the energy body is displayed as the corresponding instrument model. Simultaneously, scene images with differentiated range space configurations distinguish adjacent areas through visual features, reducing cognitive confusion for players during rapid switching. This mechanism visualizes the pitch dimension as spatial location; the movement of the player's hand in the XR visual space directly corresponds to pitch changes, achieving synchronous visualization of pitch, range, and instrument models, integrating multi-dimensional music parameters into a unified interactive dimension.

[0045] For example, the pitch-range space mapping table is as follows:

[0046] The pitch-range space mapping table divides the pitch range into five pitch ranges, corresponding to five pitch spaces: Space-Soprano, Space-High, Space-Mid, etc. Each pitch space is associated with a representative instrument type, and each pitch space corresponds to a visual theme to enhance spatial recognition.

[0047] Through the above solution, this application realizes the spatial visualization of music pitch dimension, enables the complete transmission of multi-dimensional music parameters and synchronizes interactive actions with pitch changes, avoids the problem of insufficient immersion caused by single dimension in traditional technology, and significantly improves the efficiency of human-computer interaction and the depth of music education.

[0048] In addition, traditional XR music applications lack a fine-grained representation mechanism for scales in the process of mapping pitch range in the sound space. This makes it impossible to effectively distinguish notes of different scales, making it difficult for players to understand scale changes through visual feedback, thus reducing the depth of music education and the accuracy of interaction.

[0049] In this regard, this application further proposes that each sound range space includes seven motion tracks corresponding to seven energy bodies, with each of the seven energy bodies corresponding to a different musical note and each energy body configured with a different color. In this embodiment, determining the game configuration information for the current music track may include: S310. Based on the scale-energy mapping table and all notes in the current music piece, determine the target energy information corresponding to the current music region.

[0050] In practical applications, the seven energy bodies refer to the visualized entities used to represent musical scales. They can be implemented in geometric forms such as spheres, cubes, or particle clouds, which can be specifically determined according to the corresponding target-associated musical instruments, aiming to assist players in identifying the associated musical instruments through morphological differences; the seven movement tracks can be understood as the preset linear or curved paths in the XR visual space, which can be implemented using virtual guide rails, light beam channels, or spatial grid structures, aiming to provide independent movement channels for the energy bodies of different musical scales to avoid visual confusion; the energy bodies are configured with different colors, which means that each musical scale is assigned a highly recognizable color code, which can be defined using RGB values, HSV models, or preset color palettes, aiming to establish a stable mapping relationship between musical scales and colors and enhance visual distinctiveness; the musical scale-energy body mapping relationship table refers to the data structure that stores the association rules between musical scale attributes and visual parameters, which can be implemented using database tables, hash tables, or configuration files, aiming to standardize the conversion process from musical scale information to energy body identifiers and colors.

[0051] For example, the musical scale-energy body mapping relationship table is as follows: | Musical Scale Name | Energy Body Identifier | Base Emission Color (RGB Value) | | Do C | EnergyBody_Do | (255, 0, 0) Red | | Re D | EnergyBody_Re | (255, 165, 0) Orange | | Mi E | EnergyBody_Mi | (255, 255, 0) Yellow | | Fa F | EnergyBody_Fa | (0, 255, 0) Green | | Sol G | EnergyBody_Sol | (0, 0, 255) Blue | | La A | EnergyBody_La | (75, 0, 130) Indigo | | Si B | EnergyBody_Si | (238, 130, 238) Purple |.

[0052] This application's solution subdivides the sound domain space into seven independent scale units, each corresponding to a unique motion track and energy body, forming a multi-dimensional scale representation system through a color coding mechanism. When determining game configuration information, the system analyzes the scale attributes of all notes in the current music track based on the scale-energy body mapping table, dynamically matching the corresponding energy body identifier and color parameters. When a note is triggered, the target energy body moves along the designated track, and its color characteristics reflect the scale information in real time. Players can intuitively perceive scale differences by observing the color changes of the energy body and the track position. This mechanism transforms the abstract concept of scale into a visually trackable spatial motion entity, making the hierarchical relationship of scales within the sound domain space explicit, thereby solving the technical obstacle of the lack of fine-grained scale differentiation in the sound domain space.

[0053] In one specific embodiment, when the system processes a musical piece in the key of C major, for the note "Cdo," it calls the EnergyBody_Do identifier according to the scale-energy body mapping table to generate a red energy body and deploy it on the first motion track. When the note switches to "Emi," the system automatically calls the EnergyBody_Mi identifier to generate a yellow energy body and switches to the third motion track. The energy bodies maintain their corresponding colors during movement. Players can clearly identify the "Cdo" scale by observing the red energy body moving along the first track, while the movement of the yellow energy body on the third track visually represents the "Emi" scale. The tracks are arranged vertically in parallel in the XR visual space, and the energy body color and track position form a dual scale indication. Players can understand the scale change patterns through the combination of color and spatial position feedback without needing professional music theory knowledge.

[0054] Through the aforementioned technical solution, this application establishes a fine-grained visualization framework for scales within the sound domain space, enabling highly recognizable distinctions between notes of different scales through exclusive colors and independent tracks. Players can accurately identify the current scale based on the color characteristics of the energy body and the track position, effectively avoiding interactive errors caused by scale confusion, significantly improving the completeness of musical parameter expression and the accuracy of human-computer interaction, thereby enhancing cognitive efficiency and immersive experience in the music education process.

[0055] Specifically, the lack of specific judgment criteria in the method of determining game results based on time difference makes it impossible for the system to quantify and distinguish different levels of hit accuracy, resulting in ambiguous game feedback and inaccurate player operation evaluations. This weakens the immersiveness and educational value of music games and fails to effectively improve the accuracy and fairness of human-computer interaction.

[0056] In response, this application further proposes determining the game result of the current note based on the moment of the player's most recent set gesture or any finger press action, and the target moment associated with the target energy body closest to the set judgment line, which may include: S410. Determine the time difference ΔT between the occurrence time and the target time; S420. If |ΔT| ≤ 50ms, then the game result of the current note is determined to be a perfect hit, the highest score is recorded, and the first special effect is triggered. S430. If 50ms < |ΔT| ≤ 100ms, then the game result of the current note is determined to be a hit, the base score is recorded, and the second effect is triggered. S440. If |ΔT|>100ms, then the game result of the current note is determined to be a mistake and the combo is interrupted, and no points are scored.

[0057] The timing of a set gesture or finger press refers to the time reference point at which the player sets the gesture or presses their finger. This can be determined based on the abrupt changes in bioelectrical signals detected by the player's electromyography (EMG) bracelet, or by detection from a handheld physical controller. The purpose is to accurately capture the start time of the hand movement, avoiding the latency limitations of visual or touch input. It should be noted that set gestures can be configured according to actual needs, and the specific gesture is not limited. For example, a set gesture could refer to the action of moving the hand to a corresponding energy body. The target time refers to the expected time reference point associated with the target energy body. It can be the theoretical time point when the target energy body reaches the set judgment line in its movement trajectory. Its purpose is to establish strict time alignment between the energy body's movement and the music rhythm. The time difference ΔT refers to the absolute time difference between the occurrence time and the target time. It can be calculated by the timestamp recorded by a high-precision timer. Its purpose is to provide a core indicator for quantifying the synchronization accuracy of operations. The judgment condition refers to a multi-level threshold comparison mechanism based on the time difference ΔT. It can be set to multiple continuous intervals to distinguish different accuracy levels. Its purpose is to achieve objective grading of operation evaluation. The game result includes three states: perfect hit, hit, and miss. It can be dynamically determined according to the judgment condition. Its purpose is to match operation feedback for different accuracy levels. Scoring and special effects refer to the feedback mechanism executed according to the game result. It can include score calculation logic and visual / auditory special effect triggering strategies. Its purpose is to enhance the intuitiveness and motivation of operation evaluation.

[0058] Specifically, the proposed solution first obtains the occurrence time of a set gesture or arbitrary finger press action and the target time derived from the energy body's movement trajectory, calculating the time difference ΔT between the two as a quantitative indicator. Then, ΔT is hierarchically compared with preset thresholds of 50ms and 100ms, forming a three-stage judgment interval. Finally, based on the comparison results, game result judgment, score recording, and special effect triggering are executed synchronously, thus constructing a complete chain for quantitative evaluation of operational accuracy. This process ensures strict alignment of the time reference and hierarchical execution of the judgment logic, enabling objective grading of the synchronization accuracy between player operations and music rhythm, avoiding the feedback ambiguity caused by a single judgment standard.

[0059] As a preferred embodiment, the solution of this application is implemented as follows: When the target energy body moves to the set judgment line, the system obtains in real time the occurrence time of the set gesture or arbitrary finger pressing action and the target time calculated by the energy body motion model, and calculates the time difference ΔT; if ΔT is 30ms, it is judged as a perfect hit, the highest score is recorded and a bright flashing effect is triggered; if ΔT is 70ms, it is judged as a hit, the base score is recorded and a soft halo effect is triggered; if ΔT is 150ms, it is judged as a mistake, the combo is interrupted and no score is counted. In this embodiment, the time difference threshold system is closely coupled with the game feedback mechanism, so that the evaluation of operation accuracy has a clear quantitative basis. It should be noted that the special effects corresponding to different game results can be set according to actual needs, and the specific special effects are not limited.

[0060] Through the above solution, this application can accurately quantify the synchronization accuracy between player operations and music rhythm, provide clear and graded game feedback, effectively solve the problem of ambiguous judgment, significantly improve the immersion and educational value of music games, and enhance the accuracy and fairness of human-computer interaction.

[0061] In some of the embodiments described above in this application, a dynamic switching mechanism for the sound domain space is proposed to realize the visual interaction of multi-dimensional music parameters. However, in the process of its implementation, when the target sound domain space changes abruptly, the system lacks transition prompts and targeted judgment logic, which leads to misjudgment by players due to their failure to adjust their hand positions to the new sound domain space in time. For example, the operation negligence is misjudged as a normal rhythm error, which reduces the fairness and immersion of the game.

[0062] In this regard, the music game method further proposed in this application may include: S510. When the newly determined target sound domain space is different from the current target sound domain space, a prompt message to switch sound domain space is displayed in the XR visual space during the first time period before the target energy body reaches the set judgment line.

[0063] When the newly determined target pitch range space differs from the current target pitch range space, the game result for the current note is determined based on the player's most recent set gesture or any finger press action, and the target time associated with the target energy body closest to the set judgment line. This can include: S610. After outputting the prompt message, if it is detected that the player's hand position has not been adjusted and the player's hand position at the time of the occurrence is outside the target pitch range space, then the game result of the current note is determined to be a game failure. S620. After outputting the prompt message, if it is detected that the player's hand position has been adjusted, and the player's hand position at the time of the occurrence is outside the target pitch range, then the game result of the current note is determined to be a mistake and the combo is interrupted, and no points are scored.

[0064] The first duration refers to a preset time window before the target energy body moves to the set judgment line. It can be dynamically calculated based on the energy body's movement speed and remaining distance. The purpose is to accurately match the operation window period and avoid prompting too early to interfere with the current operation or too late to cause insufficient adjustment. For example, the first duration can be 500ms.

[0065] The prompts can be understood as visual elements used to indicate the switching of sound space. They can be presented in the form of dynamic arrows, color gradients, or semi-transparent highlighted borders, with the aim of providing players with intuitive spatial movement guidance. "Hand position not adjusted" means that the player's hand does not move significantly after the prompt is output. This can be achieved by detecting the difference in hand position coordinates across consecutive frames. When the displacement is below a preset threshold, it is determined to be in a stationary state, with the aim of identifying the player's behavior of ignoring the prompt. "Hand position adjusted" means that the player's hand moves after the prompt is output. This can be determined by the change in hand position coordinates exceeding a threshold, with the aim of capturing the player's intention to actively adjust.

[0066] Specifically, the proposed solution uses the timing prediction of energy body movement trajectories to trigger visual cues before the target energy body reaches a set judgment line, providing players with a clear spatial switching warning before the operation window. Subsequently, it combines hand position adjustment status for differentiated judgment. If the hand position is not adjusted and is outside the target sound range space, the game is judged as a failure to distinguish between operational negligence and rhythm errors. If the hand position has been adjusted but is still outside the target sound range space, it is judged as an error and the combo is interrupted to acknowledge the reasonableness of minor displacement errors. This mechanism, through precise control of warning timing and refined design of judgment logic, enables the system to achieve precise alignment between operational intentions and judgment results in dynamic sound range space switching scenarios.

[0067] As a preferred embodiment, the solution of this application is implemented as follows: When the system detects that the target sound range space has switched from the low range to the high range, a dynamically flashing arrow prompt is rendered in the XR visual space, and the arrow extends along the direction of the player's hand movement; if the myoelectric bracelet detects that the hand position corresponding to the moment the finger press action occurs has not moved significantly after the prompt output and is located outside the new target sound range space, then the game is judged as a failure; if the hand position moves but does not completely enter the boundary of the new target sound range space, then it is judged as an error and the combo is interrupted, and at the same time a faint red warning light effect is displayed in the XR visual space to indicate the position deviation.

[0068] Through the above solution, this application effectively distinguishes between player operational negligence and rhythm errors during the dynamic switching of sound domain space, avoids misjudgment caused by failure to adjust hand position in time, and improves the fairness of game judgment and interactive immersion.

[0069] In some of the embodiments described above in this application, a music game method based on pitch space, energy body, and associated instruments is proposed. However, in its implementation, the game content relies entirely on predefined music tracks and lacks the ability to dynamically generate and improvise. This results in players being unable to actively participate in the music creation process and being only able to passively follow fixed sequences, which weakens the interactive depth and educational value of the game. It also fails to effectively convey the multidimensional parameters of music (such as chord progressions and pitch changes) and creative logic, limiting the immersive experience to one-way operation and lacking creative exploration.

[0070] In this regard, this application further proposes the following steps in the music game method: S710: During the game, if the set conditions are met, the AI ​​improvisation game mode is triggered. S720: In AI improvisation game mode, by setting the AI ​​model, the music segments above and below the current note of the current music piece, the chord progression of the current music piece, the music type, the target pitch range space corresponding to the current note, and the target associated instrument are processed to obtain an improvisational music segment. S730: Pause the game progress of the current music track and start the game progress of the improvised music segment.

[0071] The set conditions refer to the criteria for triggering the AI ​​improvisation game mode. These conditions can be implemented using game score thresholds, specific note sequences, or player operation modes, or other methods are not limited. The purpose is to intelligently determine the timing of improvisation intervention, avoid arbitrarily interrupting the game flow, and ensure that the improvisation segment flows naturally with the current musical context.

[0072] The AI ​​model can be understood as an algorithmic model used to generate improvisational music. It is a pre-trained, lightweight music generation model (such as a variational autoencoder, VAE). Of course, it can be implemented using a long short-term memory network (LSTM) or a Transformer architecture. Its purpose is to generate fragments that conform to the style of the original song based on the musical context information, while adapting to the interactive framework of the XR visual space. The improvisational music fragments are specifically dynamically generated musical content, which can be represented as short improvisations based on the current musical context. The purpose is to transform abstract chord progressions into operable visual elements, enhancing the player's intuitive understanding of the musical structure.

[0073] Specifically, the solution in this application monitors the fulfillment of set conditions in real time during the game. When the conditions are met, the AI ​​improvisation game mode is immediately triggered. The AI ​​model then receives the musical segments above and below the current note, chord progression, music type, target pitch range, and target associated instruments as input parameters. It uses music theory knowledge and game-specific parameters to process the data collaboratively, generating an improvisational music segment that is consistent with the style of the original song and adapted to the current XR visual space. Finally, by pausing the original game process and seamlessly switching to the improvisational segment process, a smooth transition from preset content to dynamic creation is achieved. This process makes full use of the pitch range configuration information, energy body configuration information, and associated instrument configuration information defined in the preceding claims, so that the generated improvisational segment is consistent with the existing game framework in terms of pitch distribution, visual model, and interaction logic, thereby transforming the music creation logic into operable visual feedback.

[0074] As a preferred embodiment, the solution of this application is implemented as follows: When playing a certain music piece, the system detects that the player has completed multiple perfect hits in a row and reached a preset score threshold. After meeting the set conditions, the AI ​​improvisation game mode is triggered. The AI ​​model is set to use a pre-trained Transformer network, which receives the music segments above and below the current note, the C minor chord progression, the classical music type, and the current target pitch space (corresponding to the middle pitch range) and the associated instrument (harp visual model) as input to generate a jazz-style improvisation music segment. The system then pauses the original music progress and displays the energy body of the harp visual model moving along the target pitch space track in the XR visual space. The player continues to perform finger pressing operations through the myoelectric bracelet to complete the interaction of the improvisation segment.

[0075] As another preferred embodiment, the triggering mechanism of the AI ​​improvisation game mode is as follows: the game logic module sends a trigger signal when it detects that the player has achieved a specific condition (such as hitting a complete passage perfectly in succession). Generation and Synchronization: 1. Input: The module receives the current music context, including the current pitch space, the instrument assigned to the current passage, chord progressions, tonality, etc. 2. Inference: A pre-trained, lightweight music generation model (such as a variational autoencoder VAE) generates an improvisational MIDI sequence that conforms to the music logic, conditioned on the context. To ensure real-time performance, the model inference time is strictly limited to less than 30 milliseconds. 3. Output: The output is a data pair of <improvisational audio data, interactive score data>. The interactive score data contains energy body generation instructions based on the visual attributes of the current solo instrument. Integration: During the improvisational passage, the system pauses the original score and instead drives the energy bodies based on the newly generated interactive score data, and adds special visual effects (such as Particle_Effect_ID = "Sparkle") to these energy bodies to distinguish them and enhance the reward feedback.

[0076] Through the above-mentioned solution, this application enables music games to shift from passively following predefined tracks to actively participating in improvisation. Players can experience the music generation process in real time under familiar game rules, effectively conveying multi-dimensional parameters such as chord progressions and range changes, deepening their understanding of music creation logic, significantly enhancing the interactive depth and educational value of the game, while maintaining the continuity and immersion of audiovisual feedback in the XR visual space.

[0077] In some of the embodiments described above in this application, a state switching mechanism for the target energy body is proposed to identify the current note and maintain visual continuity. However, in its implementation, the sudden change in the energy body state during note switching causes visual feedback to be interrupted, making it difficult for players to smoothly follow the music rhythm sequence, resulting in operational confusion and reduced immersion.

[0078] In this regard, this application further proposes that the music game method includes: when the target music track changes from the previous note to the next note, the target energy body corresponding to the previous note is switched from the active state to the ready state, and the target energy body corresponding to the next note is switched from the ready state to the active state; wherein, the energy body in the ready state displays a faint light of the corresponding color, and the energy body in the active state displays a bright light of the corresponding color.

[0079] In practical applications, note switching events refer to the trigger points for transitions between adjacent notes in a musical piece. These can be achieved based on precise synchronization signals of the audio timeline or MIDI event streams, with the aim of ensuring that state changes are strictly aligned with the musical rhythm. The active state can be understood as the highly salient display mode of the target energy body, which can be achieved using full-brightness rendering or dynamic lighting enhancement technology. Its purpose is to highlight the current operation focus, making it easy for players to quickly identify and execute interactions. Specifically, the preparatory state refers to the low-salient display mode of the target energy body, which can be achieved by reducing the light intensity or adding a semi-transparent gradient effect. Its purpose is to retain the visual residue of historical notes to maintain sequence continuity while avoiding interference with the current operation.

[0080] This application's solution uses note switching events as trigger conditions to drive the target energy body corresponding to the previous note from an active state to a ready state, while simultaneously driving the target energy body corresponding to the next note from a ready state to an active state. This state transition mechanism creates a dynamic visual hierarchy: the faint light of the ready state maintains the visibility of the previous note, avoiding visual interruption caused by sudden disappearance, allowing players to naturally perceive the evolutionary logic of the note sequence; the bright light of the active state instantly enhances the visual salience of the new note, guiding the player's attention precisely to the current operating area. Thus, the precise synchronization of state transitions and musical rhythm ensures the continuity of visual feedback, allowing players to smoothly follow the rhythm sequence.

[0081] As a specific implementation method, the solution of this application is implemented as follows: When the target music track switches from a C note to a D note, the EnergyBody_Do corresponding to the C note switches from an active state to a ready state, displaying a faint red light; simultaneously, the EnergyBody_Re corresponding to the D note switches from a ready state to an active state, displaying a bright orange light. This state transition is completed at the moment of note switching. The faint light of EnergyBody_Do retains the visual trace of the C note, while the bright light of EnergyBody_Re clearly indicates the timing of the D note operation, allowing the player to intuitively perceive the rhythm transition and adjust their hand movements.

[0082] Through the above solution, this application effectively solves the technical problem of visual feedback interruption when switching notes, improves the continuity of music rhythm perception and the smoothness of player operation, and avoids operation confusion and weakening of immersion caused by sudden changes in state.

[0083] In some of the embodiments described above in this application, a music game method is proposed to achieve unified modeling and interaction of multi-dimensional music parameters. However, in its implementation process, the system architecture lacks clear separation of functional modules, resulting in the coupling of configuration determination, target mapping and visual rendering, causing processing delay and audiovisual feedback disconnect, affecting the real-time performance and judgment accuracy of player operations.

[0084] In this regard, this application further proposes that the game scene of the XR visual space is divided into multiple vertical sound domain spaces.

[0085] In practical applications, XR visual space refers to a virtual visual environment constructed through extended reality technology. It can be implemented using a head-mounted display in conjunction with spatial positioning sensors, aiming to provide users with an immersive music interaction interface. Among them, the game scene refers to the visual interface layout of a music game, which can be configured as a three-dimensional virtual space to display game elements such as sound range space and energy bodies. Specifically, the vertical sound range space refers to the sound range area divided along the vertical direction. It can be implemented by dividing the visual space into multiple strip-shaped areas using vertical dividing lines, so that notes of different pitches correspond to different spatial positions at different heights, making it convenient for players to select pitches by the vertical position of their hands.

[0086] Specifically, the solution in this application decouples the division of the XR visual space game scene from the visual rendering task by dividing the sound space into multiple vertical sound domain spaces. The determination module can pre-compute the resolution of the pitch-sound domain space mapping relationship, while the rendering module focuses on driving the energy body to move within the corresponding vertical region based on the division result, avoiding the cross-calculation of configuration determination and graphics rendering. Since the vertical division method gives each sound domain space an independent vertical coordinate range, the rendering module can directly call the preset visual model to control the energy body to be displayed in a specific vertical region without waiting for the logic module to calculate the coordinates in real time, thereby shortening the response path from hand position detection to energy body movement. At the same time, the vertical division structure makes different sound domain spaces visually clearly separated, ensuring that the player's hand position detection only needs to focus on the vertical coordinate dimension, simplifying the position determination logic, and enabling the determination module to quickly verify whether the hand position is within the target sound domain space, providing a stable input for subsequent energy body movement determination.

[0087] As a specific implementation method, the solution of this application is implemented as follows: The game scene of the XR visual space is divided into seven vertical sound domain spaces, each sound domain space is distributed in a strip shape along the vertical direction, corresponding to the low range to the high range from bottom to top; wherein, the width of each vertical sound domain space is adapted to the horizontal range of the player's hand, and the height covers the vertical interval corresponding to a specific pitch range, and a faint visual dividing line is set between adjacent sound domain spaces; when the player enters the game wearing XR glasses, the system automatically activates the corresponding vertical sound domain space according to the pitch range of the current music track, the inactive area is displayed in a semi-transparent state, and the activated area clearly presents the energy body movement track and the associated instrument model.

[0088] Through the above solution, this application effectively separates the configuration determination and visual rendering functional modules, reduces the processing delay caused by multi-dimensional parameter coupling, ensures the synchronization of audiovisual feedback between energy body movement and hand position detection, and improves the real-time accuracy of player operation judgment.

[0089] In another implementation, refer to Figure 2 As shown, this application also discloses a music game device comprising: The determination module 10 is used to determine the game configuration information of the current music track; wherein, the game configuration information includes at least the target pitch range space configuration information, target energy body configuration information, and target associated instrument configuration information corresponding to the current music track; The determination module 10 is also used to determine the target pitch space and target energy body corresponding to the current note based on the game configuration information and the current note of the current music track, and the target associated instrument corresponding to the segment in which the current note is located in the target pitch space; wherein, the target pitch space is the pitch space corresponding to the current note, and the target energy body is the energy body corresponding to the current note; The rendering module 20 is used to determine that the player's hand position is within the target sound domain space of the player's XR visual space, and to control the display state of the target energy body by the visual model corresponding to the target associated musical instrument, and the target energy body moves in the target sound domain space toward the direction of the set judgment line. The determination module 30 is also used to determine the game result of the current note when the target energy body moves to the set judgment line, based on the time of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line.

[0090] In this embodiment, by systematically encapsulating the target pitch range configuration information, target energy body configuration information, and target associated instrument configuration information within a unified framework, and combining a hand position-based spatial navigation interaction mechanism with the low-latency input detection of the myoelectric bracelet, complete visualization and synchronous rendering of multi-dimensional music parameters are achieved. This effectively solves the problems of single visualization dimensions, low human-computer interaction efficiency, and audiovisual feedback disconnect in existing technologies, ensuring the complete transmission of music information and efficient human-computer synchronization, and significantly improving the user's immersive experience and interaction efficiency. Specifically, this unified modeling framework avoids the limitations of traditional solutions that rely solely on rhythm or a single pitch, enabling the collaborative processing of multi-dimensional parameters such as pitch, pitch range, scale, and instruments; the spatial navigation interaction mechanism allows users to intuitively understand pitch range changes through the matching relationship between hand position and pitch range space, replacing single-timing operations; and the low-latency characteristics of the myoelectric bracelet ensure the accurate capture of the moment of finger pressing, ensuring the real-time performance and accuracy of rhythm determination.

[0091] In another embodiment, this application also discloses a music game system, including XR glasses and an electromyography (EMG) bracelet, for implementing the above-described music game method.

[0092] In this embodiment, by combining XR glasses with an electromyography (EMG) bracelet in a hardware collaborative framework, unified modeling, synchronous visualization, and precise interaction of multi-dimensional music parameters are achieved. This solves the problems of single visualization dimension, low human-computer interaction efficiency, and disconnect between audiovisual feedback in the prior art, thereby enhancing immersion and educational depth.

[0093] Specifically, the XR glasses are configured to present an XR visual space, where the game scene is divided into multiple vertical sound domain spaces. Each sound domain space corresponds to a specific pitch range and is associated with musical instruments, displaying different scene images and energy bodies. These elements are rendered based on game configuration information, allowing the multi-dimensional parameters of the music, such as pitch, range, and scale, to be intuitively mapped to the spatial dimension. Users navigate within the sound domain space using their hand positions to understand and control the parameters. An electromyography (EMG) bracelet is configured to detect the timing of any finger press by the player, providing precise physiological signal input. Based on the timing detected by the EMG bracelet and the target timing associated with the target energy body, the game result is determined. This hardware collaboration framework ensures synchronous rendering and interactive processing of multi-dimensional parameters, avoiding a disconnect between audiovisual feedback and the XR glasses' real-time visualization output of multi-dimensional parameters, while the EMG bracelet provides low-latency interactive input. Together, they form a closed-loop system, achieving seamless integration from music parameter analysis to visual feedback.

[0094] In one exemplary embodiment, an electronic device is provided. This electronic device can be a display terminal and an input terminal in a music game system. The display terminal can be, for example, XR glasses, and the input terminal can be a myoelectric bracelet, a handheld physical controller, etc. Of course, the electronic device can also be other electronic devices within the music game system. When implementing the aforementioned music game method, this electronic device still needs to rely on information from the display terminal and the input terminal to implement the music game method. For example, when the electronic device is a display terminal, it still needs to cooperate with the input terminal and other devices to ensure the implementation of the entire music game method. Similarly, when the electronic device is another type of device, it still needs to cooperate with the display terminal and the input terminal to ensure the implementation of the entire music game method.

[0095] Among them, reference Figure 3 As shown, electronic device 100 includes at least one processor 101, memory 102, at least one network interface 104, and user interface 103. The various components in electronic device 100 are coupled together via bus system 105. It is understood that bus system 105 is used to implement communication between these components. In addition to a data bus, bus system 105 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 105 in the figure.

[0096] The user interface 103 may include a display, keyboard, or clickable electronic device (e.g., mouse, trackball), touchpad, sensor, or touchscreen. For example, in a myoelectric bracelet, the sensor or other detection unit used to detect the user's finger information can serve as the user interface 103.

[0097] It is understood that the memory 102 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 102 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0098] In some implementations, memory 102 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 1021 and application program 1022.

[0099] The operating system 1021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 1022 includes various applications, such as a media player and a browser, used to implement various application functions. Programs implementing the methods of this application embodiment can be included in the application program 1022.

[0100] In this embodiment of the application, the processor 101 executes the method steps provided in each method embodiment by calling the program or instructions stored in the memory 102, specifically the program or instructions stored in the application program 1022.

[0101] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 101. The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the processor 101. The processor 101 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 102. Processor 101 reads the information in memory 102 and, in conjunction with its hardware, completes the steps of the above method.

[0102] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing electronic devices (DSP devices, DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of this application, or combinations thereof.

[0103] For software implementation, the techniques described herein can be implemented through units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or external to the processor.

[0104] The terminal provided in this embodiment can execute all the steps of the above data processing method, thereby achieving the technical effect of the above data processing method. For details, please refer to the relevant description of the above data processing method. For the sake of brevity, it will not be elaborated here.

[0105] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0106] When one or more programs in the storage medium can be executed by one or more processors to achieve the above-described method of execution on the electronic device side.

[0107] The processor is used to execute a program stored in memory to implement the steps of the method performed on the electronic device side.

[0108] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0110] It should be noted that the terms "one implementation," "embodiment," "exemplary embodiment," and "some embodiments" used in the specification indicate that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments, whether explicitly described or not, is within the knowledge scope of those skilled in the art.

[0111] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or electronic device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or electronic device. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or electronic device that includes said element.

[0112] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. A music game method, characterized in that, The music game method includes: Determine the game configuration information for the current music track; wherein, the game configuration information includes at least the target pitch range configuration information, target energy body configuration information, and target associated instrument configuration information corresponding to the current music track; Based on the game configuration information and the current note of the current music track, the target pitch space and target energy body corresponding to the current note are determined, as well as the target associated instrument corresponding to the segment in which the current note is located in the target pitch space; wherein, the target pitch space is the pitch space corresponding to the current note, and the target energy body is the energy body corresponding to the current note; After determining that the player's hand position is within the target sound range space of the player's XR visual space, the display state of the target energy body is controlled by the visual model corresponding to the target associated musical instrument, and the target energy body moves in the target sound range space toward the direction of the set judgment line; When the target energy body moves to the set judgment line, the game result of the current note is determined based on the time of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line.

2. The music game method according to claim 1, characterized in that, The game scene in the XR visual space includes multiple sound domains, each sound domain corresponds to a multiple pitch range, and each sound domain corresponds to at least one associated musical instrument. The multiple sound domains are configured with different scene images. The game configuration information for determining the current music track includes: Based on the pitch-range space mapping table and the pitches of all notes in the current music piece, the target range space configuration information corresponding to the current music piece is determined; wherein, the pitch-range space mapping table includes the mapping relationship between multiple pitch ranges and multiple range spaces; Based on the pitch space-associated instrument mapping table and the target pitch space configuration information, the target associated instrument information corresponding to the target pitch space configuration information is determined.

3. The music game method according to claim 2, characterized in that, Each of the aforementioned sound domains includes seven motion tracks corresponding to seven energy bodies, the seven energy bodies correspond to seven musical scales, and each of the seven energy bodies is configured with a different color; The game configuration information for determining the current music track includes: Based on the scale-energy mapping table and all notes in the current music piece, the target energy information corresponding to the current music piece is determined.

4. The music game method according to claim 1, characterized in that, The determination of the game result for the current note based on the time of the player's most recent set gesture or any finger press action, and the target time associated with the target energy body closest to the set judgment line, includes: Determine the time difference ΔT between the occurrence time and the target time; If |ΔT| ≤ 50ms, then the game result of the current note is determined to be a perfect hit, the highest score is recorded, and the first special effect is triggered; If 50ms < |ΔT| ≤ 100ms, then the game result of the current note is determined to be a hit, the base score is recorded, and the second effect is triggered; If |ΔT| > 100ms, then the game result of the current note is determined to be a mistake and the combo is interrupted, and no points are scored.

5. The music game method according to claim 1, characterized in that, The music game method includes: When the newly determined target sound domain space is different from the current target sound domain space, a prompt message to switch sound domain space is displayed in the XR visual space during the first time period before the target energy body reaches the set judgment line. When the newly determined target pitch space differs from the current target pitch space, the game result for the current note is determined based on the occurrence time of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line, including: After outputting the prompt information, if it is detected that the player's hand position has not been adjusted, and the player's hand position at the time of occurrence is outside the target pitch range space, then the game result of the current note is determined to be a game failure; After the prompt message is output, if it is detected that the player's hand position has been adjusted, and the player's hand position at the time of the occurrence is outside the target pitch range, then the game result of the current note is determined to be a mistake and the combo is interrupted, and no points are scored.

6. The music game method according to any one of claims 1-5, characterized in that, The music game method includes: During the game, if the set conditions are met, the AI ​​improvisation game mode will be triggered. In the AI ​​improvisation game mode, by setting an AI model, the music segments above and below the current note of the current music piece, the chord progression of the current music piece, the music type, the target pitch range space corresponding to the current note, and the target associated instrument are processed to obtain an improvisational music segment. Pause the game process for the current music track and start the game process for the improvised music segment.

7. A music game device, characterized in that, The music game device includes: The determination module is used to determine the game configuration information of the current music track; wherein, the game configuration information includes at least the target pitch range space configuration information, target energy body configuration information, and target associated instrument configuration information corresponding to the current music track; The determining module is further configured to determine, based on the game configuration information and the current note of the current music track, the target pitch space and target energy body corresponding to the current note, and the target associated instrument corresponding to the segment in which the current note is located in the target pitch space; wherein, the target pitch space is the pitch space corresponding to the current note, and the target energy body is the energy body corresponding to the current note; The rendering module is used to determine that the player's hand position is within the target sound range space of the player's XR visual space, and then control the display state of the target energy body with the visual model corresponding to the target associated musical instrument, and the target energy body moves in the target sound range space toward the direction of the set judgment line; The determining module is further configured to determine the game result of the current note based on the time of occurrence of the player's most recent set gesture or any finger pressing action, and the target time associated with the target energy body closest to the set judgment line when the target energy body moves to the set judgment line.

8. A music game system, characterized in that, The music gaming system includes XR glasses and an electromyography (EMG) bracelet, and the music gaming system is used to implement the music gaming method as described in any one of claims 1-6.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the music game method as described in any one of claims 1-6.

10. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the music game method as described in any one of claims 1-6.

Images