Digital musical instrument audio-visual interaction system

Abstract

The invention discloses a digital musical instrument audio-visual interaction system, which belongs to the technical field of digital musical instruments and is characterized in that a player can continue to use a familiar playing mode and lower the learning threshold by providing a playing interaction terminal for a user, and then playing data of the playing interaction terminal is transmitted to data processing equipment through routing equipment; the method comprises the following steps of: acquiring audio-visual interaction data by adopting a preset data mapping rule on the basis of the playing data of the playing interaction terminal, so that a user can easily adjust control parameters without programming, the use threshold is reduced, and finally, the audio-visual interaction data is used as the basis, so that the user experience is improved. An audio-visual interaction function is provided for a user, visual feedback and sound synthesis are highly integrated, and more immersive experience is provided for a player.

Description

Technical Field

[0001] This invention belongs to the field of digital musical instrument technology, and specifically relates to a digital musical instrument audiovisual interactive system. Background Technology

[0002] Data-Driven Instruments (DDI) are a class of innovative musical instruments based on sensor input, data mapping, and digital audio processing. Their core characteristic lies in breaking through traditional physical acoustic structures and utilizing digital signals for sound synthesis and control. Currently, data-driven instruments on the market mainly involve the following key technologies: Sensor Data Acquisition: Relying on inertial measurement units (IMUs), pressure sensors, capacitive sensors, gesture tracking devices (such as Leap Motion), and biosignal sensors (EEG / EMG) to acquire human or environmental data as sound control parameters. Data Transmission and Computation: Most data-driven instruments rely on wired USB or MIDI connections for data transmission, while some high-end devices use wireless Bluetooth or Wi-Fi transmission protocols. Data Mapping and Sound Synthesis: Some systems use Max / MSP, Pure Data, and SuperCollider for data mapping and sound generation. Other devices rely on DAW plugins (such as Ableton LiveM4L) for audio synthesis and control. Machine learning (such as Wekinator) is used for gesture recognition and advanced data mapping. Although these technologies have been applied in many fields such as electronic music, interactive performances and immersive experiences, current technologies still have certain limitations.

[0003] While data-driven instruments offer new possibilities for music composition and interactive performance, existing technologies still face the following major challenges: (1) Insufficient physical form and performance intuition; Many data-driven musical instruments have completely departed from the traditional instrument form (such as the Theremin and Leap Motion controllers), resulting in unintuitive playing methods and high learning costs. Devices that rely on cameras or gesture recognition often lack stable haptic feedback, making it difficult for performers to accurately control pitch and dynamics.

[0004] (2) Limitations of traditional MIDI and wired transmission; Currently, most data-driven musical instruments on the market still rely on USB MIDI or 5-pin MIDI for data transmission. This method is limited by cable length, affecting the freedom of performance. Existing wireless MIDI solutions (such as Bluetooth MIDI) still suffer from issues such as data packet loss and unstable synchronization in low-latency real-time performance.

[0005] (3) Existing software systems are highly complex and have a high barrier to entry; Many data-driven musical instruments rely on tools such as Max / MSP, Super Collider, and Pure Data for data mapping, requiring users to have some programming skills to customize control logic. The setup and connection process for existing instruments is quite cumbersome, especially when multiple software tools are involved (such as Max / MSP combined with Ableton Live), making it difficult for novice users to get started.

[0006] (4) The existing system has weak audio and visual interactivity; The visual interaction of many data-driven musical instruments requires additional software (such as Touch Designer and Unity) for configuration, increasing operational complexity. Most existing visualization systems are separate from the sound system, lacking an integrated performance experience. Summary of the Invention

[0007] This invention provides a digital musical instrument audiovisual interactive system to solve the technical problems existing in the prior art.

[0008] A digital musical instrument audiovisual interaction system includes: a digital musical instrument device, a routing device, a data processing device, and an audiovisual interaction device; The digital musical instrument device is used to provide users with a performance interaction terminal and to collect performance data from the performance interaction terminal during the user's use of the performance interaction terminal. The routing device is used to connect the digital musical instrument device to the data processing device using an automatic configuration strategy, so as to transmit the performance data of the performance interaction terminal to the data processing device. The data processing device is used to acquire audiovisual interactive data based on the performance data of the performance interaction terminal and by adopting a preset data mapping rule. The audiovisual interaction device is used to provide users with audiovisual interaction functions based on the audiovisual interaction data, and to complete the audiovisual interaction of digital musical instruments.

[0009] Furthermore, the digital musical instrument device includes a performance interaction terminal and a data acquisition terminal; The performance interaction terminal is used to provide users with an interactive terminal for stringed instruments, so that users can perform performance interactions. The data acquisition terminal is used to collect performance data from the performance interaction terminal during the user's use of the performance interaction terminal.

[0010] Furthermore, the data acquisition terminal includes a communication module, a microprocessor, an attitude sensor, a wire displacement sensor, a thin-film potentiometer, a matrix pressure sensor, an ultrasonic sensor, and a rotary potentiometer; The communication module is used to provide communication functions for the microprocessor, so that the microprocessor can transmit the acquired performance data to the routing device; The microprocessor is used to provide data acquisition, packaging, and transmission functions; The posture sensor is used to collect posture data of the user during the use of the performance interaction terminal; The string displacement sensor is used to collect string plucking data during the user's use of the performance interaction terminal; The thin-film potentiometer is used to collect string sliding data during the user's use of the performance interaction terminal; The matrix pressure sensor is used to collect string pressure data during the user's use of the performance interaction terminal; The ultrasonic sensor is used to collect distance data between the user's hand and the interactive performance terminal during the use of the interactive performance terminal. The rotary potentiometer is used to control the working mode of the entire data acquisition terminal. Among them, the posture data, string plucking data, string sliding data, string pressing data, and distance data are collectively used as performance data.

[0011] Furthermore, the digital musical instrument device is connected to the data processing device using an automatic configuration strategy, including: The routing device provides a local area network and allows the digital musical instrument device and data processing device to access the local area network; When the routing device receives the IP address transmitted by the data processing device, it broadcasts the IP address of the data processing device to the digital musical instrument devices connected in the local area network, so that the digital musical instrument devices record the IP address of the data processing device and communicate with it.

[0012] Furthermore, the local area network is implemented via a WIFI network.

[0013] Further, transmitting the performance data from the performance interaction terminal to the data processing device includes: The system acquires data packets containing performance data from the digital musical instrument device and the interactive performance terminal, and parses the target address in the data packets; wherein the target address is the IP address of the data processing device. The data packet is transmitted to the data processing device according to the target address in the data packet.

[0014] Furthermore, the preset data mapping rules are configured by the user through the Max / MSP interface.

[0015] Furthermore, in the preset data mapping rule, the performance data and the audiovisual interaction data are mapped in a many-to-many manner.

[0016] Furthermore, the audiovisual interaction data includes lighting data, spray data, music data, and visual data.

[0017] Furthermore, the audiovisual interactive device includes a light strip, an ultrasonic atomizer, a music player, and a digital video player; The light strip is used to execute lighting data; The ultrasonic atomizer is used to execute spray data; The music player is used to execute music data; The digital image player is used to process visual data.

[0018] This invention provides a digital musical instrument audiovisual interactive system. By providing users with a performance interaction terminal, performers can use familiar playing methods, reducing the learning threshold. Then, the performance data from the performance interaction terminal is transmitted to a data processing device through a routing device, improving the portability and freedom of performance and ensuring stable and reliable data transmission. Based on the performance data from the performance interaction terminal, audiovisual interactive data is obtained using preset data mapping rules, allowing users to easily adjust control parameters without programming, further lowering the usage threshold. Finally, based on the audiovisual interactive data, audiovisual interactive functions are provided to users, with visual feedback and sound synthesis highly integrated, providing performers with a more immersive experience. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 This is a schematic diagram of the structure of a digital musical instrument audiovisual interactive system provided in an embodiment of the present invention.

[0021] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0023] To facilitate understanding of the technical solutions described in the embodiments of the present invention by those skilled in the art, a data-driven musical instrument is first introduced. A data-driven musical instrument is typically not a single entity, but a device formed by three interconnected conceptual elements: a device for acquiring input data; a software program for mapping or analyzing data; and a component capable of receiving and responding to modified data to control the sound production of musical parameters in real time. Musical Performance with Data-driven Instruments is a relatively new form of electronic music expression. The performer uses a control device—a data-driven controller—assembled from a development board, one or more sensors (such as photosensitive sensors, pressure sensors, etc.), and actuators (such as RGB LED strips, vibration motors, etc.). The performer perceives and controls the state of the sensors through senses such as touch, vision, and hearing. The resulting data is received by a computer program, which controls the generation and changes of elements such as sound and images.

[0024] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] like Figure 1 As shown, this embodiment of the invention provides a digital musical instrument audiovisual interaction system, including: a digital musical instrument device 101, a routing device 102, a data processing device 103, and an audiovisual interaction device 104; The digital musical instrument device 101 is used to provide a performance interaction terminal for the user and to collect performance data of the performance interaction terminal during the user's use of the performance interaction terminal. The performance interaction terminal can be a stringed instrument. For example, the performance interaction terminal can be set up in the form of an electric guitar so that players familiar with electric guitars can use it proficiently without any training and create new sounds using existing playing techniques.

[0026] However, a single performance interaction terminal cannot realize data-driven musical instruments. Therefore, the digital musical instrument device 101 provided in this embodiment of the invention should also be equipped with a data acquisition terminal or module to realize the acquisition of performance data from the performance interaction terminal.

[0027] The routing device 102 is used to connect the digital musical instrument device 101 to the data processing device 103 using an automatic configuration strategy, so as to transmit the performance data of the performance interaction terminal to the data processing device 103. The routing device 102 can provide a local area network for communication between the digital musical instrument device 101 and the data processing device 103, improving the portability and freedom of performance and ensuring stable and reliable data transmission.

[0028] The data processing device 103 is used to acquire audiovisual interactive data based on the performance data of the performance interaction terminal and by adopting a preset data mapping rule. Preset data mapping rules can be user-defined data mappings, allowing users to easily adjust control parameters without programming, lowering the barrier to entry and enabling users to create more types of performances.

[0029] The audiovisual interaction device 104 is used to provide audiovisual interaction functions to users based on the audiovisual interaction data, and to complete the audiovisual interaction of digital musical instruments.

[0030] Audiovisual interaction functions should include at least visual interaction functions and music interaction functions. By integrating visual and music interaction, a more integrated experience can be provided to users.

[0031] This invention provides a digital musical instrument audiovisual interactive system. By providing users with a performance interaction terminal, performers can use familiar playing methods, reducing the learning threshold. Then, the performance data from the performance interaction terminal is transmitted to the data processing device 103 through a routing device 102, improving the portability and freedom of performance and ensuring stable and reliable data transmission. Based on the performance data from the performance interaction terminal, audiovisual interactive data is obtained using preset data mapping rules, allowing users to easily adjust control parameters without programming, further reducing the usage threshold. Finally, based on the audiovisual interactive data, audiovisual interactive functions are provided to users, with visual feedback and sound synthesis highly integrated, providing performers with a more immersive experience.

[0032] In this embodiment of the invention, the digital musical instrument device 101 includes a performance interaction terminal and a data acquisition terminal; The performance interaction terminal is used to provide users with an interactive terminal for stringed instruments, so that users can perform performance interactions. The data acquisition terminal is used to collect performance data from the performance interaction terminal during the user's use of the performance interaction terminal.

[0033] In this embodiment of the invention, the data acquisition terminal includes a communication module, a microprocessor, an attitude sensor, a wire displacement sensor, a thin-film potentiometer, a matrix pressure sensor, an ultrasonic sensor, and a rotary potentiometer. The communication module is used to provide communication functions for the microprocessor so that the microprocessor can transmit the acquired performance data to the routing device 102; The communication module can be configured as an M5StickC integrated module. The M5StickC integrated module is a multi-functional development board based on the ESP32 microcontroller. It features built-in Wi-Fi and Bluetooth communication modules, and is equipped with infrared, RTC, microphone, LED, IMU, buttons, PMU, and other tools. It boasts excellent expansion interfaces and many compatible expansion modules. Its compact size makes it ideal for developing data-driven musical instruments. The M5StickC environment is set up using the Arduino IDE.

[0034] Therefore, in addition to providing stable communication functions, the communication module can also provide more expansion interfaces, providing more possibilities for the use of data acquisition terminals.

[0035] The microprocessor is used to provide data acquisition, packaging, and transmission functions; The microprocessor can be configured as an ESP32 development board. The ESP32 development board is used for IoT application development and also features Wi-Fi functionality. It has 26 GPIO interfaces and can connect multiple sensors simultaneously. Due to the need for wireless transmission, an additional 5V2A lithium battery is required, powered via the DC interface.

[0036] The posture sensor is used to collect posture data of the user during the use of the performance interaction terminal; The attitude sensor can be set to the IM948 attitude sensor. The IM948 is a small ten-axis attitude sensor module compatible with Bluetooth and serial communication. It integrates high-precision accelerometers, gyroscopes, magnetometers, and thermo-barometers, and can transmit data wirelessly via Bluetooth BLE. Therefore, the attitude sensor can effectively collect the user's posture during the performance, providing basic data for interaction.

[0037] The string displacement sensor is used to collect string plucking data during the user's use of the performance interaction terminal; The pull-wire sensor can be an industrial pulse type, with a wire length of 1000mm, a single-turn hub length of 100mm, and 400 pulses, installed in the Surmountar (i.e., the interactive performance terminal) panel. The principle of this pull-wire sensor is AB phase pulse, collector NPN signal, AB phase angle difference of 90 degrees, rectangular square wave high and low level pulses. Therefore, in the Arduino program design, two interrupt functions and one global variable are used to realize phase detection and direction determination, that is, reading the level of the B phase pulse at the rising edge of phase A; a high level indicates pulling the wire, and vice versa for retracting. In addition, when wiring, a resistor of approximately 1.5kΩ needs to be connected between phase A and 5V, and between phase B and 5V, to provide sufficient voltage and ensure stable readings from the development board.

[0038] The thin-film potentiometer is used to collect string sliding data during the user's use of the performance interaction terminal; The Surmountar fingerboard is fitted with two 500mm long thin-film potentiometers (SoftPot MembranePotentiometer – 500mm). When a finger or other object is slid across its surface, a linear impedance change is observed, making it well-suited for simulating the strings of plucked musical instruments.

[0039] When using this sensor, its data pins should be connected to the interface with pull-up resistors on the ESP32 development board. Additionally, when wiring, a resistor of approximately 1.5kΩ should be connected between the sensor data pins and the GND of the development board to ensure that the data does not fluctuate when the finger is not touching the surface of the controller.

[0040] The matrix pressure sensor is used to collect string pressure data during the user's use of the performance interaction terminal; The matrix pressure sensor, also known as a matrix flexible thin-film pressure sensor, uses the RX-M0404M model. It has 16 sensing points arranged in 4 rows and 4 columns, with a range of 1 kg. The sensor is mounted below the neck of a Surmountar violin; when pressure is sensed, the resistance decreases as the pressure increases. In this invention, this sensor is primarily used as a switch and a aftertouch controller.

[0041] The ultrasonic sensor is used to collect distance data between the user's hand and the interactive performance terminal during the use of the interactive performance terminal. The ultrasonic sensor, namely the HC-SR04 ultrasonic ranging module, is installed in the Surmountar panel and is used to measure the distance between the performer's hand and the panel.

[0042] The rotary potentiometer is used to control the working mode of the entire data acquisition terminal; that is, to control whether the data acquisition terminal is powered on and working.

[0043] Among them, the posture data, string plucking data, string sliding data, string pressing data, and distance data are collectively used as performance data.

[0044] In this embodiment of the invention, an automatic configuration strategy is used to connect the digital musical instrument device 101 to the data processing device 103, including: The routing device 102 provides a local area network and allows the digital musical instrument device 101 and the data processing device 103 to access the local area network; When the routing device 102 receives the IP address transmitted by the data processing device 103, it broadcasts the IP address of the data processing device 103 to the digital musical instrument device 101 connected in the local area network, so that the digital musical instrument device 101 records the IP address of the data processing device 103 and communicates with it.

[0045] In this embodiment of the invention, the local area network is implemented via a Wi-Fi network. The device itself does not perform calculations; data is directly transmitted to the computer via Wi-Fi, relying on the CPU for processing, ensuring low latency and high-performance computing capabilities.

[0046] This invention innovates the wireless connection method for data-driven musical instruments, proposing a bidirectional transmission and dynamic IP adaptation technology based on WIFI network, providing a more flexible, stable and convenient solution for connecting wireless musical instruments with different computers.

[0047] In existing technologies, because data transmission relies on the OSC (UDP) protocol, the sending end (instrument) typically needs to know the IP address of the receiving end (computer). However, the computer's IP address changes frequently in different network environments, leading to unstable connections for traditional OSC devices and requiring users to manually reconfigure the IP. The solution of this invention optimizes this process in the following ways: The digital musical instrument device 101 is equipped with a dedicated router that automatically connects to the Wi-Fi network upon power-on, ensuring a stable network environment for the device. The data processing device 103 simply connects to the Wi-Fi network, enters its IP address, and then broadcasts the IP address to all devices within the local area network. The digital musical instrument device 101, acting as a receiver, listens for the broadcast messages and records the IP address of the data processing device 103 as the receiving address for subsequent data. This ensures stable data transmission for the digital musical instrument device 101, eliminating the need for complex manual configuration even if the IP address of the data processing device 103 changes.

[0048] This technical solution provides a more flexible solution for connecting the development board-based wireless digital musical instrument device 101 with different data processing devices 103, enabling the device to quickly adapt to different network environments, improving the convenience and stability of the connection, and avoiding connection interruption problems caused by IP changes. This innovation enhances the usability of wireless data-driven musical instruments in performance, composition, and stage performance, and provides new ideas for future wireless interactive musical instrument systems.

[0049] In this embodiment of the invention, transmitting the performance data from the performance interaction terminal to the data processing device 103 includes: The device acquires data packets containing performance data from the digital musical instrument device 101 and the target address in the data packets is parsed; wherein the target address is the IP address of the data processing device 103. The data packet is transmitted to the data processing device 103 according to the target address in the data packet.

[0050] In this embodiment of the invention, the preset data mapping rules are configured by the user through the Max / MSP interface. Although developed based on Max / MSP, it provides a simple and intuitive UI, allowing users to easily adjust control parameters without programming. The interaction logic is smooth, and all connections and data mappings can be completed within the Max / MSP preset interface, without the need for additional configuration of multiple software tools.

[0051] In this embodiment of the invention, the performance data and the audiovisual interaction data are mapped in a many-to-many manner in the preset data mapping rule.

[0052] Unlike compositions for traditional acoustic instruments, in composing for data-driven instruments, the mapping relationship between playing actions and the audiovisual feedback of the work is not determined by physical laws, but must be established by the composer. Therefore, mapping strategies are of great importance in interactive music; by changing the mapping strategy, even if the playing method remains the same, the resulting music will have a significant change.

[0053] This invention mainly adopts a many-to-many mapping strategy, which can use multiple motion parameters to correspond to multiple MIDI parameters / effects parameters (that is, multiple data in the performance data correspond to multiple data in the audiovisual interaction data), and then use MIDI data / audio data to control the mapping switching of different music segments, while simultaneously controlling the screen parameters, so as to achieve the effect of audiovisual mutual control.

[0054] For example, in Max, sensor data can be converted into CC controller parameters using the ctlin / ctlout components, and then communicated with the host software via the from Max port, thereby achieving sound control. In the music production host AbletonLive, there are two methods: 1. Create a MIDI track with output to Max and no sound source plugins, using silent MIDI notes as triggers; 2. Mount the built-in Max for Live plugin to the track, and use the send or udpsend components to communicate with Max.

[0055] In this embodiment of the invention, the audiovisual interactive data includes lighting data, spray data, music data, and visual data.

[0056] In this embodiment of the invention, the audiovisual interactive device 104 includes a light strip, an ultrasonic atomizer, a music player, and a digital video player. The light strip is used to execute lighting data; For example, the SK6812 LED strip can be selected. This strip has a waterproof layer and is installed in a groove on the side of the Surmountar's back, allowing light to pass through a small hole at the front. Connect the strip to an M5StickC and use the Arduino FastLED library to program and drive the strip's changes. Since real-time control of the strip's changes is required, five integer variables were used in the Arduino code: H—hue [0, 255]; S—saturation [0, 255]; V—brightness [0, 255], where 0 is off; N—LED number, indicating which LED starts executing the command; Q—number of LEDs, indicating how many LEDs execute the command. By using these five variables, almost all the strip's changing effects can be achieved. Real-time control of the strip's changes can be achieved by receiving the five integers sent by Max using the UDP protocol. The specific audio-visual interaction logic can be written in Max when creating the audiovisual work.

[0057] The ultrasonic atomizer is used to execute spray data; The ultrasonic atomizer consists of a dual-head ultrasonic atomizing driver board, two atomizing plates, an atomizing bracket, and a water container. The entire assembly is fixed in a groove on the back of the Surmountar atomizer. Water mist can be emitted through small holes on the front, creating a Tyndall effect in conjunction with the LED strip. The atomizer is connected to an ESP32 development board, and the operating state of the atomizing plates is controlled via the atomizer's control pins: it operates when the pin is low and stops when the pin is high or floating.

[0058] The music player is used to execute music data; The music player can be a speaker device, and the music part can use the Max / MSP built-in sound synthesis module, with small file size (only 1MB), high running efficiency, and no need for additional computing burden.

[0059] The digital image player is used to process visual data.

[0060] The digital video player can be set up as a display screen. To match the music style, the visuals are primarily abstract, and the visuals will switch to adapt to the presets as the music / scene progresses. An audiovisual interaction plugin system can be configured, allowing users to adjust audiovisual parameters directly within the Max / MSP interface without requiring additional third-party software. Visual feedback and sound synthesis are highly integrated, providing performers with a more immersive experience.

[0061] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus 103 to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus 103, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device 103 to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing device 103, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] Those skilled in the art will understand that all or part of the steps in the above facts and methods can be implemented by a program instructing related hardware. The program or the program described therein can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: at this time, the corresponding method steps are introduced. The storage medium can be ROM / RAM, magnetic disk, optical disk, etc.

[0066] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A digital musical instrument audiovisual interaction system, characterized by, The application relates to a digital musical instrument system. The digital musical instrument device is used for providing a playing interactive terminal for a user and collecting playing data of the playing interactive terminal during use of the playing interactive terminal by the user. The routing device is used for connecting the digital musical instrument device to the data processing device by using an automatic configuration strategy to transmit the playing data of the playing interactive terminal to the data processing device. The data processing device is used for obtaining audio-visual interactive data by using a preset data mapping rule based on the playing data of the playing interactive terminal. The audio-visual interactive device is used for providing audio-visual interactive functions for the user based on the audio-visual interactive data to complete audio-visual interaction of the digital musical instrument. The digital musical instrument device comprises a playing interactive terminal and a data collection terminal.

2. The digital musical instrument audiovisual interaction system of claim 1, wherein, The playing interactive terminal is used for providing a string interactive terminal for the user to perform playing interaction. The data collection terminal is used for collecting playing data of the playing interactive terminal during use of the playing interactive terminal by the user. The data collection terminal comprises a communication module, a microprocessor, a posture sensor, a pull-wire displacement sensor, a thin-film potentiometer, a matrix pressure sensor, an ultrasonic sensor and a rotary potentiometer.

3. The digital musical instrument audiovisual interaction system of claim 2, wherein, The communication module is used for providing a communication function for the microprocessor to transmit the collected playing data to the routing device. The microprocessor is used for providing data collection, packaging and sending functions. The posture sensor is used for collecting posture data during use of the playing interactive terminal by the user. The pull-wire displacement sensor is used for collecting string plucking data during use of the playing interactive terminal by the user. The thin-film potentiometer is used for collecting string sliding data during use of the playing interactive terminal by the user. The matrix pressure sensor is used for collecting string pressing data during use of the playing interactive terminal by the user. The ultrasonic sensor is used for collecting distance data between the user and the playing interactive terminal during use of the playing interactive terminal by the user. The rotary potentiometer is used for controlling the working mode of the whole data collection terminal. The posture data, the string plucking data, the string sliding data, the string pressing data and the distance data are collectively used as the playing data. The automatic configuration strategy comprises the following steps:

4. The digital musical instrument audiovisual interaction system of claim 1, wherein, The routing device provides a local area network and allows the digital musical instrument device and the data processing device to access the local area network. When the routing device receives an IP address transmitted by the data processing device, the routing device broadcasts the IP address of the data processing device to the digital musical instrument device connected in the local area network, so that the digital musical instrument device records the IP address of the data processing device and communicates. The local area network is realized by a WIFI network.

5. The digital musical instrument audiovisual interaction system of claim 4, wherein, The playing data of the playing interactive terminal is transmitted to the data processing device by the following steps:

6. The digital musical instrument audiovisual interaction system of claim 1, wherein, ​ Acquire a data packet of performance data of a performance interactive terminal transmitted by the digital musical instrument device, and parse a target address in the data packet; wherein the target address is an IP address of a data processing device; According to the target address in the data packet, transmit the data packet to the data processing device.

7. The digital musical instrument audiovisual interaction system of claim 1, wherein, The preset data mapping rule is configured by a user through a Max / MSP interface.

8. The digital musical instrument audiovisual interaction system of claim 7, wherein, In the preset data mapping rule, the performance data and the audio-visual interactive data are mapped in a many-to-many form.

9. The digital musical instrument audiovisual interaction system of claim 1, wherein, The audio-visual interactive data includes light data, spray data, music data and visual data.

10. The digital musical instrument audiovisual interaction system of claim 9, wherein, The audio-visual interactive device includes a light strip, an ultrasonic atomizer, a music player and a digital image player; The light strip is used for executing the light data; The ultrasonic atomizer is used for executing the spray data; The music player is used for executing the music data; The digital image player is used for executing the visual data.

Images