Online performance interaction method and system based on VR technology, and storage medium

Abstract

The invention discloses an online performance interaction method and system based on a VR technology, and a storage medium, and the system comprises a virtual performance scene module which is used for building a VRChat virtual performance world and providing a virtual performance scene for a DJ user; the VRChat virtual performance world can be watched and interacted by audience users; the DJ interaction control module is used for synchronizing the performance performed by the user through the real DJ controller to the VRChat virtual performance world; and the audio playing module is used for playing DJ user playing or / and external input audio in the VRChat virtual performance world. Through the gesture input module, the virtual disk drive synchronized with the real DJ controller, the audio playing module and the like, a user can obtain an immersive performance feeling in a virtual world.

Description

Technical Field

[0001] The present invention belongs to the field of VR technology, and in particular relates to an online performance interaction method and system, and a storage medium based on VR technology. Background Art

[0002] With the rapid development of virtual reality technology, social VR platforms have become a crucial alternative to traditional offline performances. Existing research suggests that the anonymity of avatars can effectively reduce gender bias and stereotypes in social interactions. A virtual DJ culture has emerged on platforms like VRChat, but it faces two major technical bottlenecks. First, existing virtual DJ systems (such as TribeXR) lack support for physical device interaction, resulting in a lack of tactile feedback for professional DJs, hindering their ability to demonstrate their expertise and limiting creative expression. Second, the inadequate digital reproduction of non-verbal communication mechanisms in traditional virtual performances makes it difficult to establish an emotional connection between DJs and audiences.

[0003] Prior art Chinese patent application CN202210819362.8 discloses a method for remote interactive performances based on XR, which belongs to the field of content production, and includes the following steps: A1: making a virtual background; A2: planning the area to be spliced; A3: XR rendering of the main venue and the branch venue; A4: shooting real-life virtual scenes for the main venue, and shooting real-life virtual scenes for the L screen; A5: mixing the picture of the branch venue camera with the voice of the singer in the branch venue, and encoding and transmitting it; A6: the main venue receives the stream from the branch venue, decodes it, and then splices and synthesizes the picture; A7: VR implants the spliced ​​video, and then renders and outputs it for playback.

[0004] The main purpose of the above-mentioned existing technologies is to achieve interactive performances in different locations. However, they do not provide a virtual DJ system, and are unable to retain the tactile feedback and operating habits of professional DJs, thus limiting creative expression. Summary of the Invention

[0005] The purpose of the present invention is to provide an online performance interaction method and system, and a storage medium based on VR technology, which partially solves or alleviates the above-mentioned deficiencies in the prior art and is a virtual performance system that can support real device interaction and enhance the sense of presence.

[0006] In order to solve the above-mentioned technical problems, the present invention specifically adopts the following technical solutions: A first aspect of the present invention is to provide an online performance interaction system based on VR technology, comprising: A virtual performance scene module is used to build a VRChat virtual performance world to provide a virtual performance scene for DJ users; the VRChat virtual performance world can be viewed and interacted with by audience users; DJ interactive control module, used to synchronize the user's performance through real DJ controller to the VRChat virtual performance world; The audio playback module is used to play DJ user playback and / or external input audio in the VRChat virtual performance world.

[0007] Furthermore, the virtual performance scene module includes: A virtual stage is a platform for DJ users to perform virtual performances, which can be watched by audience users from multiple angles; and audience users can interact with the DJ users' performances through motion capture devices.

[0008] The virtual disc player is a virtual image of a physical disc player, which can simulate the status of the physical disc player according to the signals transmitted by the DJ interactive control module; Several DJ avatars for DJ users to choose as their avatars to perform on the virtual stage; DJ voice,receives the sound signal from the audio playback module and plays it on the virtual stage.

[0009] Furthermore, the virtual performance scene module also includes virtual social facilities for DJ users and / or audience users to conduct virtual social activities.

[0010] Furthermore, the virtual performance scene module also includes a waiting room for DJ users to select DJ virtual images during performances; the waiting room is provided with a virtual disc player and a virtual mirror for users to practice and observe their own performances.

[0011] Furthermore, it also includes a sign-in board set in the waiting room, where users can sign in to obtain control of the virtual disc player on the virtual stage.

[0012] Furthermore, the DJ interaction control module includes: The DJ controller input module is used to receive MIDI signals from the DJ controller, copy the received MIDI signals, and send them to the audio playback module and the virtual disc player in the virtual performance scene module through the music production visualization module port and the virtual disc player port respectively; the virtual disc player synchronizes with the real DJ controller according to the MIDI signals; The gesture input module is used to recognize the user's gestures and control the movements of the DJ virtual image in sync with the hand gestures through VRChat navigation.

[0013] Furthermore, the audio playback module includes: The music production visualization module is used to output the user played audio to the virtual audio mixer module; The virtual audio mixer is used to mix the received user playback audio with the external input audio and output it to the virtual performance scene module as DJ voice.

[0014] Furthermore, the music production visualization module is Recordbox software.

[0015] Furthermore, the online performance interaction system also includes a media module, and the virtual performance scene module also includes a virtual video wall; the media module is used to transmit the virtual performance to the virtual video wall and / or a third-party platform.

[0016] The present invention also provides an online performance interaction method based on VR technology, comprising: Building a VRChat virtual performance world to provide a virtual performance scene for DJ users; the VRChat virtual performance world can be viewed and interacted with by audience users; Synchronize the user's performance using a real DJ controller to the VRChat virtual performance world; and play the DJ user's playback and / or external input audio in the VRChat virtual performance world.

[0017] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a computer, the computer executes the above method.

[0018] Beneficial effects: This invention utilizes a gesture input module, a virtual disc player synchronized with a real DJ controller, and an audio playback module to enable users to experience an immersive performance experience within the virtual world. A virtual video wall displays diverse content, complemented by audio and lighting effects, creating a realistic performance atmosphere and enhancing the audience's sense of immersion. It offers a wealth of personalized options, such as diverse DJ avatars and customizable virtual stage scenes, allowing DJs to showcase their unique style. The music production visualization module and virtual audio mixer support creative mixing and audio creation, meeting diverse creative needs and providing DJs with a broad creative space.

[0019] The DJ controller input module ensures accurate transmission of real-world controller signals to the virtual world. The virtual player synchronizes with the physical controller status, making DJ operation natural and smooth, reducing learning curves, maintaining operational habits, and improving performance quality. Virtual social facilities provide an interactive platform for DJs and audiences, enabling real-time communication, collaborative performances, and other social activities. Live streaming services expand audience participation, and interactive features on third-party platforms foster interaction between audiences and DJs, and between audiences themselves, fostering a vibrant social atmosphere.

[0020] The virtual video wall can play online live broadcasts and video content, showcasing performance scenes and enriching visual elements. The audio playback module supports mixing multiple audio sources, adding different sound effects to the performance, making the performance content more rich and diverse.

[0021] Based on the OBS live streaming service, performance scenes and content can be transmitted to virtual screens and third-party platforms in real time, breaking geographical restrictions, attracting global audiences, expanding the influence of performances, and tapping into potential commercial value.

[0022] Technical integration between modules, such as MIDI signal transmission and audio and video synchronization, ensures stable system operation. At the same time, the modular design facilitates functional expansion and maintenance, supporting the continuous development of the system.

[0023] This invention provides a more professional and creative solution for virtual performances, promotes technological progress and content innovation in the virtual performance industry, attracts more users to participate in virtual performances, and promotes the prosperity of the virtual performance industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the embodiments or the description of the prior art. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the various elements or parts are not necessarily drawn according to the actual scale. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without inventive work.

[0025] Figure 1 This is a schematic structural diagram of embodiment 1 of the present invention; Figure 2 This is a flow chart of signal distribution in embodiment 1 of the present invention; Figure 3 This is a flowchart of virtual signal forwarding in embodiment 1 of the present invention; Figure 4 This is a flowchart of a virtual performance in the first embodiment of the present invention; Figure 5 This is a flowchart of the audio module workflow in Example 1 of the present invention; Figure 6 This is a schematic diagram of the interface of the DJ controller (real disc player) in the first embodiment of the present invention; Figure 7 Schematic diagram of the key-value mapping between the DJ controller and the virtual disc player in the first embodiment of the present invention; Figure 8 This is a schematic diagram of MIDI mapping distribution code in Example 1 of the present invention; Figure 9 This is a schematic diagram of the MIDI settings of the Rekordbox software in Example 1 of the present invention; Figure 10 This is a schematic diagram of the virtual stage; Figure 11 This is a schematic diagram of the waiting room; Figure 12 This is a diagram of a virtual drive. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] Herein, suffixes such as "module," "component," or "unit" used to represent elements are only used to facilitate description of the present invention and have no specific meaning. Therefore, "module," "component," or "unit" may be used interchangeably.

[0028] As used herein, terms such as "upper," "lower," "inner," "outer," "front," "back," "one end," and "the other end" indicate positions or locations based on those shown in the accompanying drawings. These terms are intended solely to facilitate and simplify the description of the present invention and are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0029] As used herein, unless otherwise expressly specified or limited, the terms "installed," "provided with," and "connected" should be understood broadly. For example, "connected" may refer to a fixed connection, a detachable connection, or an integral connection; it may refer to a mechanical connection, a direct connection, an indirect connection via an intermediate medium, or internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention on a case-by-case basis.

[0030] As used herein, "and / or" includes any and all combinations of one or more of the associated listed items.

[0031] Herein, "plurality" means two or more than two, ie, it includes two, three, four, five, etc.

[0032] Example 1: like Figure 1 As shown, this embodiment provides an online performance interaction system based on VR technology, including: (1) A virtual performance scene module, which uses Unity and UdonSharp to build a VRChat virtual performance world to provide DJ users with a virtual performance scene; the VRChat virtual performance world can be viewed and interacted with by audience users.

[0033] Unity is a widely used real-time 3D development platform with powerful graphics rendering capabilities, a rich plugin library, and cross-platform functionality. It can create high-quality virtual scenes, character models, and special effects, providing realistic visuals for virtual performances. UdonSharp, a programming tool based on Unity, simplifies the process of creating interactive content in VRChat environments, allowing developers to write scripts in C# and implement features such as character movement interaction and scene element control. By combining Unity and UdonSharp, developers can efficiently build VRChat virtual performance worlds that meet the needs of virtual performances.

[0034] This virtual performance world features specialized performance scenarios designed for DJs. For example, a virtual stage serves as the core area for DJ performances, outfitted with avatars of professional equipment such as a virtual deck. The virtual deck simulates the state of real DJ controllers based on signals transmitted by the DJ interactive control module, allowing DJs to operate their equipment in the virtual environment just as they would in real life. Several DJ avatars are also available for DJs to choose from, allowing them to select the appropriate avatar based on their preferences and performance style.

[0035] Furthermore, the virtual performance world is open to audiences for viewing and interaction. Audiences can access the VRChat platform through VR devices or PCs to watch the performance. This allows viewers to observe the DJ's performance from multiple angles, enhancing the experience's immersion and enjoyment. Furthermore, the virtual performance scene module features audience interaction areas and features. For example, audiences can use motion capture devices to dance to the music, engage in social activities, and interact with the DJ and other audience members, enhancing the overall interactivity and participation of the performance.

[0036] Specifically, the virtual performance scene module includes: (1) Virtual stage such as Figure 10 As shown, the platform for DJ users to perform virtual performances can be viewed from multiple angles by audience users; and audience users can interact with the DJ users' performances through motion capture devices.

[0037] The virtual stage provides DJs with a virtual space for their musical performances, serving as the core area of ​​the entire performance. It transcends the physical limitations of a traditional stage and offers DJs an innovative approach to performance. Audiences can watch the DJ's performance from multiple angles on the dance floor. This multi-perspective viewing experience enhances audience engagement and immersion, allowing them to choose the optimal viewing position based on their preferences, creating a feeling of being in the real performance venue.

[0038] The audience can dance to the music using motion capture devices, creating a deep interaction between the audience and the performance. The motion capture devices can translate the audience's real movements into actions in the virtual environment in real time, allowing the audience to participate in the performance in their own way, enhancing the fun and interactivity of the performance.

[0039] The virtual stage is equipped with a virtual disc player, which simulates the status of a real disc player according to the signal transmitted by the DJ controller, allowing the DJ to perform professional music mixing and performance operations in the virtual environment just like in reality.

[0040] In some embodiments, there is also a virtual mirror diagonally above the virtual stage to help DJ users observe the virtual environment, audience behavior, and their own performance in all directions, so as to adjust their personal movements and mobilize the atmosphere.

[0041] (2) Virtual drive such as Figure 12 As shown, the virtual image of a physical player can simulate the state of a physical player based on signals transmitted by the DJ interactive control module. Specifically, the virtual player in this embodiment is a virtual player model based on a Pioneer DDJ-1000, set on a virtual stage. The virtual player model includes a control panel for adjusting the position and size of the virtual player to align with a physical player in reality.

[0042] The Virtual DJ, a virtual representation of a physical controller, bridges the gap between real and virtual environments. By recreating the form and functionality of the Pioneer DDJ-1000 1:1, DJs can seamlessly transfer their real-life skills to the virtual environment without retraining, achieving smooth performances.

[0043] The control panel's calibration function ensures that the virtual player perfectly matches the physical device in terms of position and size. Combined with the VR headset's spatial positioning, this gives the DJ the illusion of directly controlling the virtual equipment. This spatial anchoring effect greatly enhances the sense of presence, making it feel like they are truly at the center of the virtual stage.

[0044] The virtual player's real-time response to physical manipulations, such as turntable rotation and fader movement, forms a complete perception-action cycle. DJs can not only see the visual effects of their actions in the virtual world, but also experience a heightened sense of accomplishment through feedback from the virtual environment's lighting, shadows, and particle effects, creating a positive incentive for immersive creative expression.

[0045] (3) Several DJ avatars for DJ users to choose as avatars to perform on the virtual stage. DJ avatars provide a way for DJs to showcase their style and personality. Different DJs can choose a suitable avatar based on their preferences, performance themes, or personal branding. For example, a DJ specializing in electronic music might choose a technological or futuristic avatar; while a DJ specializing in retro music might opt ​​for a vintage-inspired avatar, creating a unique personal identity on the virtual stage and enhancing the recognition of their performances.

[0046] (4) DJ voice, which receives the sound signal from the audio playback module and plays it on the virtual stage.

[0047] DJ Voice enables the DJ's voice output on the virtual stage. The audio playback module precisely transmits the DJ's music and microphone input, such as live commentary and interactive conversations, to the virtual stage scene. This seamlessly integrates the sound with the DJ's avatar performance, allowing the audience to hear music and voice that aligns with the performance on the virtual stage, enhancing the performance's realism and coherence, making the entire show more vivid and complete.

[0048] (5) Waiting room Figure 11 As shown, a DJ user selects a DJ avatar for a performance; the waiting room is equipped with a virtual disc player and a virtual mirror for users to practice and observe their performance. A sign-in board is also provided in the waiting room, allowing users to sign in and gain control of the virtual disc player on the virtual stage.

[0049] Waiting Room allows DJs to choose their own avatars for their performances, allowing them to position and express themselves based on the theme, style, and personal preferences of their performances. This diverse selection of avatars allows DJs to better blend into the performance, creating a unique stage presence that adds a personal touch to their performances while also enhancing their confidence and engagement.

[0050] The virtual deck in the waiting room provides DJs with an opportunity to practice their skills before their actual performance. DJs can use this time to familiarize themselves with the virtual deck's feel and practice complex mixing techniques in advance, ensuring a smoother, more precise performance and minimizing mistakes. This helps improve the quality of DJ performances and provides a practical space for DJs to explore and innovate their performances in a virtual environment.

[0051] The virtual mirror allows DJs to observe their own movements and performances in real time. By observing, DJs can adjust and optimize their body movements and facial expressions to better suit the atmosphere and style of the performance.

[0052] The sign-in board effectively regulates performance flow and order. It provides clear rules for obtaining control of virtual decks, ensuring that only DJs who have signed in can operate them at any given time. This avoids the chaos of multiple DJs vying for control, ensuring that performances proceed smoothly as planned and making the entire event more organized and efficient. From a system management perspective, the sign-in board records DJ user sign-in information, which helps facilitate performance statistics and analysis.

[0053] After a successful sign-in, the DJ will be automatically transferred to the virtual stage.

[0054] (6) Virtual social facilities for DJ users and / or audience users to conduct virtual social activities.

[0055] Specifically, it includes: interactive props, including dice, chess and cards, turntables, table games, table sports, etc.; additional functions of virtual rooms, allowing users to set up private places, prevent other users from entering, mute and prevent other users from seeing the inside of the room, etc.; themed game spaces, including virtual bars, swimming pools, chess and card rooms, table sports rooms, open-air restaurants, KTVs, private cinemas, general empty rooms, etc.

[0056] Virtual social facilities break down the barriers between audience members and DJs, and between each other, that exist in traditional performances. During a performance, audience members can use these facilities to interact with the DJ in real time, sending emojis and text messages to express their enjoyment of the performance or offer suggestions. The DJ can then respond promptly. This two-way interaction transforms the audience from passive spectators into active participants, enhancing their sense of engagement and experience. Furthermore, audience members can exchange their feelings and opinions about the music, creating a lively social atmosphere and enhancing the overall enjoyment of the performance.

[0057] (7) Virtual video wall: DJ users can play online live broadcasts or videos by entering links on the control panel of the virtual LED video wall; they can also broadcast performance scenes and content.

[0058] DJ users can enter links to play online live streams or videos, greatly enriching the content sources for virtual performances. In addition to traditional music performance videos, other types of live streams, such as interviews and behind-the-scenes footage, can also be introduced, allowing viewers to not only enjoy the music but also gain more contextual information about the performance. For example, broadcasting a DJ's creative process live between performances can enhance audience understanding and interest in DJ creation.

[0059] Of course, the virtual stage performance can also be displayed on a virtual video wall, similar to a concert, providing audiences with a wider range of perspectives. Just as the large screen on the stage allows viewers to see close-ups and different angles of the performance, the virtual video wall can display details of the virtual stage, such as DJ movements and special effects. This compensates for the lack of a full view of the stage due to viewing angle limitations, enhancing the audience's viewing experience and allowing them to fully appreciate the performance.

[0060] (2) DJ interactive control module, used to synchronize the user's performance through the real DJ controller to the VRChat virtual performance world.

[0061] The module can receive real-time operation signals from real DJ controllers, such as the rotation of turntables, movement of faders, clicks of buttons, etc., and synchronize these operations to the virtual disc player and related equipment in the VRChat virtual performance world.

[0062] The DJ interactive control module specifically includes: (1) A DJ controller input module, which is used to receive MIDI signals from the DJ controller, copy the received MIDI signals, and send them to the audio playback module and the virtual disc player in the virtual performance scene module through the music production visualization module port and the virtual disc player port respectively; the virtual disc player is synchronized with the real DJ controller action according to the MIDI signals.

[0063] A DJ controller is a device specifically designed for DJ performances, allowing the DJ to control and mix music in real time to create a unique musical experience. In this embodiment, the DJ controller can be a real disc player or a simulated disc player designed specifically for this system, similar to a game joystick. The DJ controller is connected to the host computer running this system via a USB-A port and serves as the source of MIDI signal input to the system. The DJ controller input module is responsible for receiving MIDI signals from the DJ controller and then copying the received MIDI signals so that the same operation signal can be sent to the audio playback module and the virtual disc player in the virtual performance scene module at the same time to achieve synchronization of audio and virtual scenes.

[0064] Synchronize the user's performance using a real DJ controller to the VRChat virtual performance world; and play the DJ user's playback and / or external input audio in the VRChat virtual performance world.

[0065] For example, if a DJ presses the Left CUE button on a turntable to play the music on the left track back to the CUE, the system will process the user's input as follows: (1) DJ user registration and cloud verification are completed; (2) The DJ user presses the CUE-L button on the player ( Figure 6 ); (3) The disc player sends MIDI signals; (4) The signal enters the MIDI signal distribution module: (4.1) MIDI signal distribution module receives MIDI signal ( Figure 7 ): (4.2) The signal is a Control Change (CC) signal and is therefore filtered through the MIDI signal filter. (4.3) The module sends the complete information to both virtual MIDI ports (Rekordbox and VRChat); For MIDI signals output to VRChat virtual ports: (1) The signal is sent to the VRChart world; (2) The disc player update script processes the signal according to the Databook ( Figure 8 ); (3) The CUE button of the disc player in the VRChat world lights up; (4) Update status and synchronize it to the cloud.

[0066] For MIDI signals output to the rekordbox virtual port: (1) Rekordbox reads data from the virtual port; (2) According to the Mapping, read the control operation (Cue set / Play Cue Back) ( Figure 9 ); (3) The music jumps to the Cue; (4) Changes in the music input to the virtual microphone; (5) Changes in the music played in the VRChat world.

[0067] Therefore, the DJ's real-world controller operations, such as adjusting the volume, switching tracks, using special effects, etc., can be reflected in real time on the virtual disc player, allowing the virtual performance to accurately simulate the process of the real performance and present realistic visual effects to the audience.

[0068] As the DJ manipulates the physical controller, the generated MIDI signals are transmitted to the VRChat virtual world, causing the virtual turntables, faders, knobs, buttons, and performance pads to synchronize in real time. This allows the DJ to experience a virtual environment that's close to the real-life performance, as if performing on a real stage. For example, when the DJ spins the physical turntables, the virtual turntables will also rotate in sync. This high degree of consistency greatly enhances the realism of the operation, increasing the DJ's immersion and performance.

[0069] From the audience's perspective, the virtual deck is a synchronized representation of the physical DJ controller, allowing them to more intuitively understand the DJ process. The audience can see the state changes of the virtual deck device, enhancing the performance's enjoyment and understanding. For example, when the DJ adjusts the volume on the physical fader, the virtual deck's fader moves synchronously, allowing the audience to clearly see the volume changes and better appreciate the dynamic effects of the music, enhancing the audience's viewing experience.

[0070] Figure 2 It shows the overall forwarding process of signal distribution synchronization between virtual and real devices. Figure 3 and Figure 4 The specific process of forwarding and synchronizing virtual signals to Rekordbox software and VRChat virtual performance scenes is demonstrated: The user operates the DJ controller to output MIDI signals through the physical port.

[0071] MIDI signals are replicated and distributed to two virtual ports. The first is the Rekordbox virtual port, the port for the music production visualization module. MIDI signals are input into the Rekordbox client, processed, and output to the virtual audio mixer. This, combined with voice input, ultimately outputs the performance audio through the virtual microphone. The second is the VRChat virtual port, the virtual player port. MIDI signals are input into the virtual player plug-in script, triggering virtual player status updates and synchronizing the virtual player with the physical controller.

[0072] In some embodiments, during a performance, when a DJ performs operations on a real player, MIDI signals are output through the physical port. However, a DJ may perform multiple operations on the real player in succession. Therefore, if these operations are not differentiated during copying, the virtual DJ avatar's movements on the virtual player may become out of sync. Therefore, to avoid this problem or reduce the likelihood of this problem occurring, in this embodiment, a single DJ action on the real player is used as a copy node. That is, when copying MIDI signals, the MIDI signal corresponding to that action is copied, rather than all MIDI signals within the entire time period.

[0073] Specifically, sensors can be set up to detect the user's movements on a real-world player, such as sensor-equipped gloves worn by DJs while operating the player. The placement of the sensors within these gloves can be referenced in the HIGGlove VR gesture recognition gloves. The system then performs motion recognition based on the data detected by the sensors (this is prior art and will not be further described here). This allows the system to obtain the MIDI signal corresponding to each movement simultaneously with the movement recognition result. Therefore, only the MIDI signal corresponding to each movement needs to be copied and distributed, rather than copying MIDI signals within a specific timeframe. Of course, when MIDI signals corresponding to multiple movements are copied, they are automatically sorted based on the timestamp of each movement. However, the sorting of the acquired data is inevitably subject to errors or inaccuracies. Therefore, it is preferable to sort based on the start timestamp (such as the timestamp corresponding to Note On) and the end timestamp (such as the timestamp corresponding to Note Off) of each movement. For example, if the end timestamp of movement A1 is less than the start timestamp of movement B1, this indicates that movement A1 precedes movement B1, and therefore the MIDI signal corresponding to A1 is sorted before the MIDI signal corresponding to B1. For example, if the start timestamp of an action A2 is greater than the end timestamp of another action B2, it means that action A2 comes after action B2. Therefore, the MIDI signal corresponding to A2 is sorted after the MIDI signal corresponding to B2, and so on. The received MIDI signals are then spliced ​​and distributed in action units based on the sorting.

[0074] (2) Gesture input module, which is used to recognize the user's gestures and control the movements of the DJ virtual image synchronously with the hand gestures through VRChat navigation.

[0075] This module enables users to control the DJ avatar's movements through natural gestures, eliminating the need for complex keyboard controls or specialized controllers. For example, users can use common gestures like waving, nodding, and extending their hands to make the avatar greet, bow, and operate virtual equipment. This interactive method is more intuitive and intuitive, reducing user learning curves and improving the convenience and fluidity of interaction.

[0076] In this embodiment, the gesture input function supported by Steam Link and SteamVR is used as the input method of VRChat; the upper limb body movement tracking and performance in the system are achieved through hand gestures.

[0077] Steam Link is a tool for streaming PC games and other content to other devices, such as mobile devices, while SteamVR provides software support for virtual reality experiences. Steam Link and SteamVR enable gesture input. In VRChat, gesture input becomes a key interaction method, allowing users to interact naturally with the virtual environment through gestures, such as waving and grabbing. This makes user interaction in the virtual world more intuitive, reduces operational complexity, and enhances user immersion and engagement in VRChat.

[0078] Upper limb motion tracking through hand gestures makes the user's performance within the system more realistic and vivid. In a virtual performance, the DJ user can control the avatar's upper limb movements through hand movements, such as adjusting virtual disc player parameters or performing specific hand gestures. This combination of upper limb motion tracking technology and virtual performances provides DJs with a more expressive performance and a more immersive viewing experience for the audience. The audience can see the DJ avatar's upper limb movements synchronized with their actual hand movements, enhancing the credibility and enjoyment of the virtual performance.

[0079] (3) Audio playback module, used to play DJ user playback and / or external input audio in the VRChat virtual performance world.

[0080] The audio playback module receives audio played by the DJ user, as well as external audio input, and plays it within the VRChat virtual performance world. This imbues the virtual performance world with sound elements, providing a richer sensory experience for users. Whether it's a DJ's own unique musical composition or various sound effects associated with the performance, this module can be used to play, immersing the audience in the musical atmosphere.

[0081] As the DJ avatar operates the virtual disc player, the audio playback module plays the corresponding music and sound effects, allowing the audience to see the avatar's movements synchronized with the audio, creating a realistic performance. This synchronized audio and visual experience makes the virtual performance more vivid and realistic, enhancing the audience's viewing experience.

[0082] Figure 5 The audio playback module's specific program flow is shown. For audio controlled by the disc player and output through Rekordbox, it is input into a virtual microphone created by the system, along with the user's microphone, at a volume percentage that can be set by the user. VRChat then sets the secondary microphone as the user's voice, allowing the performance audio to be played in VRChat while also preserving the user's voice. The audio playback module specifically includes: (1) Music production visualization module, used to output the user played audio to the virtual audio mixer module.

[0083] The Music Production Visualization module is responsible for outputting user-played audio to the Virtual Audio Mixer module, ensuring that the audio created or played by the user is accurately processed by subsequent modules. The music played by the DJ using a real DJ controller on the virtual stage is transmitted to the Virtual Audio Mixer through this module for further mixing and output.

[0084] This module's visualization features provide users with intuitive feedback on music production. By presenting audio data in a visual format, such as waveform display and spectrum analysis, it helps DJs understand the characteristics and changes of the audio in real time. In this embodiment, the music production visualization module is Recordbox software.

[0085] (2) A pseudo-audio mixer, which is used to mix the received user-played audio with the external input audio and output it to the virtual performance scene module as DJ voice.

[0086] The virtual audio mixer receives user-generated audio from the music production visualization module and external audio input and mixes them together. This feature enables diverse audio combinations, making the audio of virtual performances richer and more three-dimensional. For example, DJ music can be mixed with external audio such as live ambient sounds and special effects to create a unique audio atmosphere and enhance the immersive experience of the performance.

[0087] (4) Media module, used to transmit virtual performances to a virtual video wall and / or a third-party platform.

[0088] With OBS live streaming, virtual performances are no longer constrained by physical space. Audiences in different cities or countries can watch the performance in real time as long as they have an internet connection.

[0089] This module supports real-time streaming of performance content to third-party platforms. This allows DJ performances to be displayed not only on virtual screens but also live-streamed on popular platforms like YouTube and TikTok. By leveraging the widespread influence and massive user base of third-party platforms, virtual performances can reach a wider audience and attract attention from users across multiple platforms.

[0090] like Figure 1 As shown, in terms of hardware, the virtual performance scene module in this embodiment is deployed on VRChat; the DJ controller input module in the DJ interactive control module is deployed on the DJ controller, and the gesture input module is deployed on the head display; and the audio playback module and media module are both deployed on a PC, i.e., a personal computer.

[0091] Example 2: This embodiment also provides an online performance interaction method based on VR technology, including: S1 builds a VRChat virtual performance world to provide a virtual performance scene for DJ users; the VRChat virtual performance world can be viewed and interacted with by audience users; S2 obtains the performance of the user through the real DJ controller, synchronizes it to the VRChat virtual performance world, and plays the DJ user's playback and / or external input audio in the VRChat virtual performance world.

[0092] Furthermore, the method further comprises the steps of: Generate a number of DJ avatars for DJ users to select as avatars to perform on a virtual stage; Receives sound signals from the audio playback module and plays them on the virtual stage.

[0093] In some embodiments, the step of obtaining a user's performance using a real DJ controller and synchronizing it to the VRChat virtual performance world specifically includes: Receive MIDI signals from the DJ controller, copy the received MIDI signals, and send them to the audio playback module and the virtual disc player in the virtual performance scene module respectively, so that the virtual disc player synchronizes with the real DJ controller according to the MIDI signals; Recognize user gestures and control the movements of the DJ avatar in sync with the VRChat navigation and hand gestures.

[0094] Example 3: This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the method described in the second embodiment.

[0095] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0096] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a computer terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0097] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. An online performance interactive system based on VR technology, characterized by include: The virtual performance scene module is used to build the VRChat virtual performance world to provide DJ users with virtual performance scenes; The VRChat virtual performance world can be viewed and interacted with by audience users; DJ interactive control module, used to synchronize the performance of DJ users using real DJ controllers to the VRChat virtual performance world; Audio playback module, used to play DJ user playback and / or external input audio in the VRChat virtual performance world; Wherein, the virtual performance scene module includes: A virtual stage is a platform for DJ users to perform virtual performances, which can be viewed from multiple angles by audience users. Audience users can also interact with the DJ users' performances through motion capture devices. The virtual disc player is a virtual image of a physical disc player, which can simulate the status of the physical disc player according to the signals transmitted by the DJ interactive control module; Several DJ avatars for DJ users to choose as their avatars to perform on the virtual stage; DJ voice,receives the sound signal from the audio playback module and plays it on the virtual stage.

2. The online performance interactive system based on VR technology according to claim 1 is characterized in that The virtual performance scene module also includes virtual social facilities for DJ users and / or audience users to conduct virtual social activities.

3. The online performance interactive system based on VR technology according to claim 1, characterized in that: The virtual performance scene module also includes a waiting room for DJ users to select DJ virtual images during performances; the waiting room is provided with a virtual disc player and a virtual mirror for users to practice and observe their own performances.

4. The online performance interactive system based on VR technology according to claim 2, characterized in that: The invention also includes a sign-in board arranged in the waiting room, and the sign-in board is used for users to sign in to obtain control of the virtual disc player on the virtual stage.

5. The online performance interactive system based on VR technology according to claim 4, characterized in that: The DJ interactive control module includes: The DJ controller input module is used to receive MIDI signals from the DJ controller, copy the received MIDI signals, and send them to the audio playback module and the virtual disc player in the virtual performance scene module through the music production visualization module port and the virtual disc player port respectively; the virtual disc player synchronizes with the real DJ controller according to the MIDI signals; The gesture input module is used to recognize the user's gestures and control the movements of the DJ virtual image in sync with the hand gestures through VRChat navigation.

6. The online performance interactive system based on VR technology according to claim 1 is characterized in that The audio playback module includes: The music production visualization module is used to output the user played audio to the virtual audio mixer module; The virtual audio mixer module is used to mix the received user playback audio with the external input audio and output it to the virtual performance scene module as DJ voice.

7. The online performance interactive system based on VR technology according to claim 6, characterized in that: The music production visualization module is Recordbox software.

8. The online performance interactive system based on VR technology according to claim 1 is characterized in that The online performance interaction system further includes: a media module, and the virtual performance scene module further includes a virtual video wall; the media module is used to transmit the virtual performance to the virtual video wall and / or a third-party platform.

9. An online performance interaction method based on VR technology, characterized in that: An online performance interaction system based on VR technology as described in any one of claims 1 to 8, wherein the online performance interaction method comprises: Building a VRChat virtual performance world to provide a virtual performance scene for DJ users; the VRChat virtual performance world can be viewed and interacted with by audience users; Acquire the user's performance using a real DJ controller and synchronize it to the VRChat virtual performance world; and play the DJ user's playback and / or external input audio in the VRChat virtual performance world; The virtual performance scene includes: A virtual stage is a platform for DJ users to perform virtual performances, which can be viewed from multiple angles by audience users. Audience users can also interact with the DJ users' performances through motion capture devices. The virtual disc player is a virtual image of a physical disc player, which can simulate the status of the physical disc player according to the signals transmitted by the DJ interactive control module; Several DJ avatars for DJ users to choose as their avatars to perform on the virtual stage; Receive sound signals from the audio playback module and play them on the virtual stage; The steps for obtaining the user's performance using a real DJ controller and synchronizing it to the VRChat virtual performance world include: Receive MIDI signals from the DJ controller, copy the received MIDI signals, and send them to the audio playback module and the virtual disc player in the virtual performance scene module respectively, so that the virtual disc player synchronizes with the real DJ controller according to the MIDI signals; Recognize user gestures and control the movements of the DJ avatar in sync with the VRChat navigation and hand gestures.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a computer, cause the computer to perform the method according to claim 9 .

Images