Music production device and method for operating a music production device

Abstract

A music production device (100) comprises a user interface (110) that is configured to receive user inputs from a user and to convert the user inputs into digital signals, and a control unit (150) that is configured to receive the digital signals and to generate music data based on the received digital signals The user interface (110) comprises at least one vision based tactile sensor (120) that comprises a transparent gel (130) on or in which markers (135) are placed and an image sensor (140) that observes the markers (135) through the transparent gel (130). User inputs are received by receiving a pressing action of the user on the transparent gel (130) that deforms the gel such that positions of the markers (135) change. During the pressing action the image sensor (140) is configured to capture image data of the change of positions of the markers (135), which image data allow deduction of physical parameters of the pressing action. The digital signals are generated from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action. The music data are generated according to the physical parameters of the pressing action represented by the digital signals.

Description

MUSIC PRODUCTION DEVICE AND METHOD FOR OPERATING A MUSIC PRODUCTION DEVICEFIELD OF THE INVENTIONThe present disclosure relates to a music production device and a method for operating a music production device.BACKGROUDModem music production relies to a large extent on digital music production and / or conversion devices. For example, music production devices or sampling devices such as the Akai MPC provide platforms for users to play and record music in an intuitive manner by hitting or pressing with fingers on a few buttons or touch pads that serve as interface. In particular, it is possible to mix, arrange, and modify samples of existing music (ranging e.g. from tones of different instruments, over melodies or compositions to orchestral music) or sound samples in order to create new music files / data.SUMMARY OF THE INVENTIONAlthough such devices are widely used, they can be improved with regard to their interfaces and the possible options for developing music.The present disclosure mitigates this shortcoming of conventional music production devices.To this end, a music production device is provided that comprises a user interface that is configured to receive user inputs from a user and to convert the user inputs into digital signals, and a control unit that is configured to receive the digital signals and to generate music data based on the received digital signals. Here, the user interface comprises at least one vision based tactile sensor that comprises a transparent gel on or in which markers are placed and an image sensor that observes the markers through the transparent gel. User inputs are received by receiving a pressing action of the user on the transparent gel that deforms the gel such that positions of the markers change. During the pressing action the image sensor is configured to capture image data of the change of position of the markers, which image data allow deduction of physical parameters of the pressing action. The digital signals are generated from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action. The music data are then generated according to the physical parameters of the pressing action represented by the digital signals.Further, a method for operating a music production device as described above is provided. The method comprises: receiving as user input a pressing action of a user on the transparent gel that deforms the gel such that positions of the markers change; with the image sensor, capturing image data of the change of position of the markers, which image data allow deduction of physical parameters of the pressing action; converting the pressing actions into digital signals by generating the digital signals from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action; and receiving the digitalsignals at the control unit and generating by the control unit music data according to the physical parameters of the pressing action represented by the received digital signals.These music production devices differ therefore from known music production devices by the use of vision based tactile sensors as user interface. These tactile sensors use a temporal series of images of a deformable, transparent portion, i.e. of a transparent gel, in or on which markers are placed, to deduce the pressure / force acting on the transparent gel from the observed movement of the markers in / on the gel. Compared to conventional tactile sensors such as capacitive or resistive-based tactile sensors, such vision based tactile sensors can achieve higher spatial and temporal resolution, a more precise and reliable parametrization of the received pressure / force, and a reduction of latency, while keeping the structure of the sensor comparably simple. Moreover, since vision based tactile sensors operate in principle with arbitrarily shaped transparent gels, it is possible to provide the music production devices with user interfaces of various forms that are only difficultly providable for conventional tactile sensors, such as strongly curved or even spherical surfaces. This allows bringing music production devices into forms that cannot be provided for known music production devices.BRIEF DESCRIPTION OF THE DRAWINGSFig. 1 is a simplified block diagram of a music production device;Figs. 2A and 2B are simplified block diagrams of a vision based tactile sensor;Fig. 3 is a simplified illustration of physical parameters characterizing a pressing action;Figs. 4A and 4B are simplified illustration of a music production device;Fig. 5 is a simplified illustration of a control of music generation;Fig. 6 is a simplified illustration of a spherical music production device;Fig. 7 schematically shows a process flow for operating a music production device.Fig. 1 shows a schematic illustration of a music production device 100. The music production device 100 can be used to generate digital music data that can be reproduced via speakers of an (arbitrary) music reproduction device.To this end, the music production device 100 comprises at least a user interface 110 and a control unit 150. The user interface 110 is configured to receive user inputs from a user via at least one vision based tactile sensor 120 and to convert the user inputs into digital signals.The at least one vision based tactile sensor 120 comprises at least a transparent gel 130 on or in which markers 135 are placed and an image sensor 140 that observes the markers 135 through the transparent gel 130, and theuser inputs are received by receiving a pressing action of the user on the transparent gel 130 that deforms the transparent gel 130 such that positions of the markers 135 change.The concept of vision based tactile sensors 120 is in principle known, and an in-depths description of such sensors can be omitted here. Accordingly, only a short review of the most important properties of the used vision based tactile sensor 120 will be given here.As indicated above and as shown in Fig. 1 (see in particular the schematic cross section in the lower part of Fig. 1), the vision based tactile sensor 120 is essentially constituted by a transparent gel 130 comprising markers 135 and an image sensor 140 such as a camera. The gel 130 is easily deformable under the typical pressure providable by one or several finger(s) / finger tip(s) of a user. However, it resists deformation such that by pressing on it with the fingers or the hand, it is not possible for a typical human to deform it permanently. This means that after a pressing action has ended, the transparent gel 130 will resume its original form. In particular, also the markers 135 included in the transparent gel 130 will return to their original positions after the end of a pressing action. The transparent gel 130 is moreover transparent in the wavelength band observable by the image sensor 140, e.g. in (part of) the visible spectmm or in the infrared spectrum. Outside the wavelength bands used for observing the markers 135, the transparent gel 130 may have any color. Various materials for the transparent gel 130 are possible, as long as it is deformable, elastic and at least partially transparent. Possible materials would include but are not limited to standard gels, polymer gels (such as polyvinyl alcohol, polyacrylic acid, polyacrylonitrile and polyvinyl chloride gels), elastomers, and rubber.Thus, the gel 130 can be easily deformed by a user, which leads to a change of the positions of the markers 135. Since the gel 130 is transparent, this change of positions can be easily observed by the image sensor 140.The markers 135 can be randomly distributed through the transparent gel 130, as shown in Fig. 1 or can be arranged according to a specific order, for example in a regular grid. The markers 135 may be arranged at a surface of the transparent gel 130 (e.g. by placing them on the transparent gel 130 after manufacturing the transparent gel 130) and / or within the transparent gel 130. The markers 135 can also have a random shape. They may have a structure that allows determining their orientation in space, such as schematically illustrated in Fig.1. However, they may also be spherical or even dot like for the spatial resolution of the image sensor 140. The markers 135 may also be deformable such that not only their location, but also their form changes during a deformation of the transparent gel 130. The markers 135 may be fluorescent at a wavelength (band) observable by the image sensor 140. In principle, any markers 135 can be used that have a sufficiently high contrast with the surrounding transparent gel 130 to allow identification from the image data obtained via the image sensor 140. For example, the markers 135 may be printed (for example using industrial ink) directly on the surface of the transparent gel 130 or on another thin material glued or stuck on the surface of the transparent gel 130 (or as a layer within the transparent gel 130). The markers 135 may be illuminated by illumination means (not shown) such as LEDs in a wavelength band observable by the image sensor 140. The illumination means may also emit light that excites fluorescence in a wavelength band observable by the image sensor 140.As shown in the lower part of Fig. 1, the image sensor 140 may be located beneath the transparent gel 130, i.e. opposite to the side of the transparent gel 130 on which the user input is received, i.e. opposite to the surface ofthe transparent gel 130 that receives a pressing action. However, in principle the image sensor 140 may also be located at a side of the transparent gel 130, i.e. arranged horizontally adjacent to the transparent gel 130 for a vertical pressing action.The image sensor 140 may be separated by a spacer 142 from the transparent gel 130 and optics 144 may be used to project the image of the markers 135 of the transparent gel 130 onto the image sensor 140. The optics 144 may be merely constituted by a pinhole in an otherwise non-transparent spacer 142. However, also any kind of lens system embedded into or constituted by the spacer 142 is conceivable. Further, the image sensor 140 may also be placed directly onto the transparent gel 130 or may be separated from it by a transparent spacer 142 without any optics. The placement of the image sensor 140 with respect to the transparent gel 130 and the use / nature of the optics 144 depends in principle only on the intended field of view Vi of the image sensor 140, which defines the sensitive area of the vision based tactile sensor 120. Typically, the closer the image sensor 140 is to the transparent gel, the higher the spatial resolution of the images of the transparent gel 130 will be, while the observed region becomes smaller.During a pressing action on the transparent gel 130, the image sensor 140 is configured to capture image data of the change of position of the markers 135, which image data allow deduction of physical parameters of the pressing action. As explained above, the transparent gel 130 is fully elastic in the force regime applied to it during normal operation, i.e. during operation via the fingers of a human. Thus, the original position of the markers at the start of any pressing action is the same. Moreover, also the deformation of the transparent gel 130 for two identical pressing actions will be identical. This allows deducing from the deformation of the transparent gel 130 the nature of the pressing action that led to the deformation, i.e. the physical parameters of this pressing action. The deformation is in turn obtainable from the changing positions of the markers 135 during the pressing action. Accordingly, the image data obtained by the image sensor 140 allow deducing the physical parameters of the pressing action, such as e.g. the position of the pressing action on the transparent gel 130, the strength and direction of the involved force, and the like.Figs. 2A and 2B show schematically the results of a pressing action on the transparent gel 130. Fig. 2A shows a schematic side view of the transparent gel 130 in uncompressed state. Markers 135 are provided along the surface of the transparent gel 130. Fig. 2B shows the effect of a pressing action A on the transparent gel 130. The pressing action A pushes the markers 135 at the position where the pressing action A hits the transparent gel 130 downwards. Adjacent markers 135 are shifted sidewards, and in part upwards. More distant markers 135 are (almost) not affected. The pressing action A provides therefore a particular change of markers 135 that can be used as a footprint to identify the according pressing action A.The image sensor 140 may comprise a plurality of pixels 141 as also illustrated in Figs. 2A and 2B. The image sensor 140 may be an event-based vision sensor that is capable to generate event data for each of said pixels 141, whenever an intensity detected by one of the pixels 141 is changed by an amount that is larger than an event detection threshold. Such event-based vision sensors are thus configured to detect changes in the intensity of the received light caused e.g. by motion. They can operate asynchronously, i.e. not frame based, and report a change of intensity above the event detection threshold immediately, thereby reducing the latency compared to a conventional sensor. In a vision based tactile sensor intensity changes are only generated due to the shiftingpositions of the markers 135 during pressing actions. The markers 135 are (comparably) sparsely distributed in the transparent gel 130 and constitute the only relevant source of information. Thus, observing only intensity changes caused by the markers instead of capturing entire image frames every frame cycle reduces the amount of redundant information. This reduces in turn the energy consumed by the sensor and the computing power necessary to deduce information about the deformation of the transparent gel 130 from the image data. Accordingly, using an event-based vision sensor as image sensor 140 of the vision based tactile sensor 120 may provide several advantages.Nevertheless, it is of course also possible to use an image sensor 140 that is capable to generate image frames indicating the intensity of light received at each pixel, and to deduce deformations of the transparent gel 130 from these frames.In both cases, the spatial resolution of the images captured during deformation of the transparent gel 130 is sufficiently high to allow a deduction of the position of the pressing action with a resolution that is higher than the resolution obtainable by typical capacitive or resistive-based tactile sensors.The image data obtained from the image sensor 140 are converted into digital data and may constitute the digital signals provided by the user interface 110. In particular, the digital data may be the event data or the image frames generated by the above described examples for the image sensor 140. Additionally or alternatively, the user interface 110 may comprise processing logic that processes the image data such as to generate digital signals that differ from the image data, e.g. by extracting and / or encoding relevant parameters from these image data. For example, a deformation of the transparent gel 130 during a pressing action may be reconstructed from the image data, and the physical parameters of the pressing action may be determined from the reconstructed deformation. The determined physical parameters may then be encoded as digital data.However, the nature of the digital signals generated by the user interface is in principle arbitrary, as long as the digital signals represent the physical parameters of the pressing action, i.e. as long as these physical parameters are deducible from the digital signals. Deduction of the physical parameters of the pressing action from the observed deformation may be done according to a theoretical model of the transparent gel 130, by numerical simulation of the behavior of the transparent gel 130, by generating a look-up table during a calibration and testing phase of the music production device 100, via an artificial intelligence model that is trained accordingly, or the like.Here, it should be noted that the user interface 110 may comprise next to the vision based tactile sensor(s) 120 also other input means 112 such as e.g. conventional buttons, controls, keyboards, a (touch) screen or the like. The inputs made on these input means 112 before, during or after operating the vision based tactile sensor(s) 120 may influence the form of the digital signals. For example, it may be possible to switch on and off functions of the music production device 100 via the input means 112 that rely on different physical parameters such as the contact area of the pressing action, a shear force, a shift of a position of the pressing action during the pressing action, or the like. Then, according to the functions that are switched on and off, digital signals provided from the user interface 110 may contain only information on the physical parameters that are necessary for these functions. However, it is also possible that the information contained in the digital signals is independent fromthe user input received at the input means 112 and that such user inputs only affect the operation of the control unit 150.The control unit 150 is configured to receive the digital signals from the user interface. Based on the received digital signals, and in particular based on the physical parameters of the pressing action represented by the digital signals, the control unit 150 generates music data. For example, if the user interface 110 merely provides the image data as digital data to the control unit 150, the control unit 150 may be configured to reconstruct a deformation of the transparent gel 130 during a pressing action from the image data, such as e.g. event data or image frames, to determine the physical parameters of the pressing action from the reconstructed deformation, and to generate the music data according to the determined physical parameters. If the digital data already contain the physical parameters of the pressing action, the control unit 150 may use these data as direct input for the generation of the music data. Further, also commands received via the input means 112 of the user interface 110 may influence the generation of the music data, allowing or disallowing for example usage of specific physical parameters for the music data generationHere, it should be noted that in principle any manner of generating the music data based on the deducible physical parameters of the pressing action is conceivable, as long as reception of pressing actions with different physical parameters lead to different music data. The music production device 100 may operate for example as an electronic instmment, where the manner and / or sequence how one touches the user interface 110 determines the generated melody or composition. The music production device 100 may also operate as a sampler or mixer that allows modifying and / or rearranging different samples of music depending on how one touches the user interface 110. The music production device 100 may also combine these operation modes and / or may be switchable between such modes. The music data that is generated by the control unit 150 of the music production device may be in any uncompressed or compressed format, such as e.g. WAV, AIFF, PCM, FLAC, WMA, MP3, AAC, MIDI, or the like.The control unit 150 may be any software or hardware component that can carry out the functions of the control unit 150 described herein. For example, the control unit 150 may be any processing device such as processor, a CPU, a computer, an FPGA, an ASIC or the like. The control unit 150 may also be an application or program running on such a processing device. The control unit 150 may be implemented in hardware, in software or as a mixture of both.As shown in Fig. 1, the music production device 100 may comprise a sound reproduction unit 160 that comprises loudspeakers and that is configured to reproduce the music data generated by the control unit 150. This means that the music production device 100 can directly output the music generated by it. Accordingly, the music production device 100 may function as a music instrument. Additionally, the music production device 100 may still output the music data to other devices for further processing or reproduction.An advantage of the above-described music production device is that the vision based tactile sensor 120 used as main part of the user interface 110 allows a very precise and complete deduction of the physical parameters of the pressing action over time in both magnitude and direction. This is illustrated in Fig. 3.Fig. 3 shows a three-dimensional force vector F caused by a pressing action and acting on the transparent gel 130. This force vector F can be decomposed into components being parallel to the surface of the transparent gel 130 (Fx and Fy), and a component acting vertically on the transparent gel 130 (Fz). The decomposition into parts Fx and Fy is arbitrary and can be chosen freely, e.g. to be perpendicular to each other and / or to be parallel to boundaries of the transparent gel 130. Each of the components Fx, Fy, Fz of the force vector F, or the entire force vector F may constitute a physical parameter of the pressing action that is recorded by the user interface 110 and / or the control unit 150 and used as input for music data generation. Here, the parallel components Fx, Fy may be used in combination as a shear force S = Fx + Fy acting in parallel to the surface of the transparent gel 130.Further, it is also possible to determine the position O at which the pressing action hits the transparent gel 130, and a contact area C of the object exerting the pressing action, like e.g. one or several finger(s) of a user of the music production device 100. The position O of the pressing action can e.g. be the center of gravity of the contact area C. Further, the contact area C and the vertical force Fz may be used to determine the pressure P on the transparent gel 130. Also, a pressure distribution in the contact area C may be determined, e.g. by determining pressure components perpendicular to the deformed surface of the transparent gel 130.Of all these parameters the temporal development may be determined and also used in controlling the music data generation such that reaching two times the same absolute value of pressure and / or force leads to different music data, when the temporal development that led to said absolute was different.By varying the above physical parameters, a series of inputs is provided to the control unit 150 that is used by the control unit 150 to generate corresponding music data. In this manner the music production device 100 can be controlled based on the variation and / or adjustment of different parameters such as a real instrument.Figs. 4A and 4B show two possible implementations of the music production device 100. As shown in Fig. 4A the user interface 110 may comprise one vision based tactile sensor 120 that is virtually divided into a plurality of input regions 121. Such a large area vision based tactile sensor 120 may be obtainable by placing the image sensor 140 comparably far from the transparent gel 130. With this configuration, while physically there is only one transparent gel 130 and one image sensor 140 that is able to observe the entire transparent gel 130, different regions of the transparent gel 130 are treated as different input regions 121. This means that pressing actions received in these different regions will lead to generation of different music data. The vision based tactile sensor 120 is therefore virtually separated into a plurality of input channels. The shape of the input regions 121 may here be freely adjustable and may also be changed dynamically.Alternatively or additionally, as shown in Fig. 4B, the user interface 110 comprises a plurality of vision based tactile sensors 120 that constitute a plurality of input regions 121. Thus, instead of virtually separating a single vision based tactile sensor 120, it is also possible to provide smaller, physically distinct vision based tactile sensors 120 as input regions 121. This allows a higher spatial resolution within each vision based tactile sensor 120, since the image sensors 140 of the vision based tactile sensors 120 can be placed closer to the respective transparent gels 130 than in the case of one large vision based tactile sensor 120.In both cases, the control unit 150 is configured to determine from the received digital signals in which of the input regions 121 a pressing action has been received and to generate for pressing actions received at different input regions 121 different music data Ml, M2 that cause, when reproduced, different audible perceptions for a typical human.Thus, a user of the music production device 100 can generate different music data by pressing on different input regions 121 just as when playing a music instrument.As shown in Fig. 1, the music production device 100 may comprise a memory 170 for storing a plurality of samples as original music data. Here, at least some of the samples may represent different tones at different pitches, played by one or several instruments and / or sang by one or a plurality of humans, i.e. the samples reproduce the sound of a music instrument or several (also different) music instruments or the sound of singing of a soloist or a choir. Additionally or alternatively, at least some of the samples may represent different parts of a piece of music, i.e. a melody or composition or parts of it may be stored in digital format. Further, also various sounds may be recorded in the memory 170 as some of the samples. The control unit 150 is then configured to generate the music data from such original music data.In particular, when receiving a user input on one of the input regions 121 the control unit 150 retrieves a corresponding sample / corresponding original music data from the memory 170. Which sample is associated with which input region 121 can be freely selected, e.g. via the input means 112 or via an external computer connected to the music production device 100. From the various original music data retrieved based on a sequence of user inputs on the input regions 121 the control unit 150 generates the music data to be output. In this manner the music production device 100 can be used as a sampler / mixer of different music samples.In addition to integrate the original music data of the sample into the generated music data according to the operated input region 121, the control unit 150 is configured to determine the physical parameters of pressing actions and to vary the original music data based on the determined physical parameters. This means that the manner the vision based tactile sensor(s) 120 are operated influences the manner in which the samples are integrated into the final music data. Thus, hitting an input region 121, for example, hard and fast leads to music data that will sound differently, when reproduced, than music data that are generated when hitting the same input region 121 soft and slowly, although in both cases the same original music data are used as a basis. This can be compared to playing a real music instrument, where the manner one plays a piano keyboard or a guitar string determines the sound of the instrument, although the pitch of the tone is the same.Further examples how the physical parameters of the pressing action influence or modify the sample associated with the respective input region 112 are: Based on the length of the pressing action, a duration of the audible perception represented by the original music data is changed, e.g. by increasing the duration for a longer pressing action. Thus, just as for a real music instrument, the longer an input region 112 is pressed, the longer will the respective sample be played, once the respectively generated music data are reproduced. If a sample is constituted by a tone of a certain pitch or by a sound, pressing the respective input region 112 determines the time duration of the tone or sound in a replay of the respectively generated music data.Further, based on the strength of a pressure of the pressing action a volume of the audible perception represented by the original music data is changed, preferably by increasing the volume for larger pressures. Thus, pressing with much strength on the input region 112 will lead to a louder replay of the sample than actuating the input region 112 weakly. This is comparable to a piano keyboard, where a strong pressure leads to louder tones. Here, also the temporal development of the pressure may have an influence on the generated music data / the replay of the sample. For example, only if the pressure is increased sufficiently quickly on the input region 112 an input will be recognized, while a slowly increasing pressure will be disregarded, again in analogy to a piano, where no sound is generated, if the keyboard is operated too slowly. Further, the strength of the pressure of the pressing action may also have different effects than determining a loudness of the sample in the generated music data. The pressing strength may also determine a shift of the original tone of the sample, a replay speed or the like.Further, based on a movement of a position of the pressing action a pitch and / or a timbre of the audible perception represented by the original music data may be changed. For example, when moving a finger on the input region 112, the sample will be modified that is associated with the position of the first touch of the finger. Preferably the pitch can be changed e.g. by raising it for a movement in one predetermined direction and lowering it for a movement in the opposite direction. Similarly, the timbre can be changed by moving in another direction and can be changed back with a movement back to the position of the first touch.The above is schematically illustrated in Fig. 5. Fig. 5 shows the vision based tactile sensor 120, its transparent gel 130 and the markers 135. On the transparent gel 130 contact areas C are indicated. A larger contact area C will be associated with larger pressure. Thus, by changing the size of the contact area C, the volume can be changed. Further, when dragging the contact point along the arrow marked with X, the pitch can be lowered or raised. Similarly, when dragging the contact point along the arrow marked with Y, the timbre can be changed. Of course, all these actions can be carried our concurrently. Further, the above-described effects and their triggers may also be interchanged. For example, volume may be changed by moving along the arrow marked with X, while the timber is changed depending on the exerted pressure and the pitch is controlled by moving along the arrow marked with Y.Also, it has to be emphasized that the above manners of controlling the generation of music data according to the deducible physical parameters are only examples. Music data generation can for example also be based on the number of fingers / the shape of the contact area, such that different shapes lead to different types of sound. Further, gestures might be executed on the transparent gel 130 such as pinching or stretching the transparent gel 130, which could lead to specific sound effects.In this manner, the intended replay of a sample can be fine tuned with a high precision. Also, a completely new class of music instmments with new playing rules can be constructed in this manner.Although depicted in the schematic illustration of Figs. 4A and 4B to be flat, the transparent gel 130 may have any form as long as this form can withstand bending forces due to gravity. This is due to the fact that all that is needed for the vision based tactile sensor 120 is that the transparent gel 130 and its markers 135 return to their original shape / position once a user interaction with the vision based tactile sensor 120 has been ended. Then, any deformation can be deduced from the changing positions of the markers 135. The position changes of themarkers 135 can, however, be observed for any shape of the transparent gel 130. This allows forming the transparent gel 130 in a highly flexible manner that can e.g. be customized to specific users or that can be used to provide music production devices 100 of various shapes.In particular, the surface of the transparent gel 130 that receives the pressing action may be curved more than it is possible for a conventional tactile sensor. For example, the curvature may be such that a deviation of the surface from a flat surface is more than 1 cm. This means, when (virtually) putting the curved surface that receives the pressing action on a flat surface, the curved surface is at least once positioned more than 1 cm away from the flat surface.This flexibility of design of the vision based tactile sensors 120 is schematically exemplified in Fig. 6. Fig. 6 shows a music reproduction device 100 that is formed as a sphere. The surface of the sphere is completely (or almost completely up to ports for connecting to external devices and / or the electrical grid) covered with vision based tactile sensors 120 that constitute different input regions 112. This music production device 100 can be held by a user in his hands. By rotating the music production device 100 or by moving his hands along the surface of the music production device 100, the user can play the different input regions 112 distributed on it. Thus, the flexible shape of the vision based tactile sensors 120 allow designing totally new shapes for music instruments.As indicated above, the music production device 100 may be provided with its own loudspeakers. Also, the music production device 100 may be a portable device that is powered by a battery. Just the same, the music production device 100 may need an external loudspeaker and an external power supply. In this regard, it is also conceivable that the control unit 150 is arranged in a component that differs from the component carrying the user interface 110, and that the digital signals are transmitted wire-bound or wirelessly from the user interface 110 to the control unit 150. Then, the component carrying the control unit 150 may be equipped with a sound reproduction device and may reproduce the music data that are generated based on the digital signals in realtime.The above-described operation of the music production device 100 can be summarized in the schematic method illustrated in Fig. 7. At SI 10 a pressing action of a user is received as user input on the transparent gel 130 that deforms the transparent gel 130 such that positions of the markers 135 change. At S120 the image sensor 140 captures image data of the change of positions of the markers 135, which image data allow deduction of physical parameters of the pressing action. At S130 the pressing action is converted into digital signals by generating the digital signals from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action. At S140 the digital signals are received at the control unit 150 which generates music data according to the physical parameters of the pressing action represented by the received digital signals.In this manner it is possible to provide a new type of music production device 100 that is highly customizable and that allows music generation / sampling / mixing with great flexibility and high precision.The present technology can also be configured as described below:[1] A music production device (100) comprising a user interface (110) that is configured to receive user inputs from a user and to convert the user inputs into digital signals; and a control unit (150) that is configured to receive the digital signals and to generate music data based on the received digital signals; wherein the user interface (110) comprises at least one vision based tactile sensor (120) that comprises a transparent gel (130) on or in which markers (135) are placed and an image sensor (140) that observes the markers (135) through the transparent gel (130); user inputs are received by receiving a pressing action of the user on the transparent gel (130) that deforms the gel such that positions of the markers (135) change; during the pressing action the image sensor (140) is configured to capture image data of the change of positions of the markers (135), which image data allow deduction of physical parameters of the pressing action; the digital signals are generated from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action; and the music data are generated according to the physical parameters of the pressing action represented by the digital signals.[2] The music production device (100) according to [1], wherein the physical parameters represented by the digital signals comprise at least a position (O) of the pressing action on the transparent gel, a three-dimensional force vector (F) generated by the pressing action, a pressure (P) and / or a pressure distribution on the transparent gel caused by the pressing action, a shear force (S) on the transparent gel caused by the pressing action, a size and / or shape of a contact area (C) at which the vision based tactile sensor is contacted during the pressing action, and a temporal development of any of said parameters.[3] The music production device (100) according to [1] or [2], wherein the user interface (110) comprises one vision based tactile sensor (120) that is virtually divided into a plurality of input regions (121); and / or the user interface comprises a plurality of vision based tactile sensors (120) that constitute a plurality of input regions (121); and the control unit (150) is configured to determine from the received digital signals in which of the input regions (121) a pressing action has been received and to generate for pressing actions received at different input regions (121) different music data that cause, when reproduced, different audible perceptions for a typical human.[4] The music production device (100) according to [3], further comprising a memory (170) for storing a plurality of samples as original music data; wherein the control unit (150) is configured to generate the different music data from different original music data such that each piece of the different music data causes, when reproduced, an audible perception that is based on the respective sample.[5] The music production device (100) according to [4], whereinat least some of the samples represent different tones at different pitches, played by one or several instruments and / or sang by one or a plurality of humans; and / or at least some of the samples represent different parts of a piece of music; and / or at least some of the samples represent different sounds.[6] The music production device (100) according to [4] or [5], wherein the control unit (150) is configured to determine the physical parameters of pressing actions and to vary the original music data based on the determined physical parameters.[7] The music production device (100) according to [6], wherein based on the length of the pressing action a duration of the audible perception represented by the original music data is changed by increasing the duration for longer pressing action; and / or based on the strength of a pressure of the pressing action a volume of the audible perception represented by the original music data is changed, preferably by increasing the volume for larger pressures; and / or based on a movement of a position of the pressing action a pitch and / or a timbre of the audible perception represented by the original music data is changed, preferably by raising the pitch for a movement in one predetermined direction and lowering the pitch for a movement in the opposite direction.[8] The music production device (100) according to any one of [1] to [7], wherein the image sensor (140) is an event-based vision sensor that comprises a plurality of pixels (141) and is capable to generate event data for each of said pixels (141), whenever an intensity detected by a pixel (141) is changed by an amount that is larger than an event detection threshold; the digital data are the event data; and the control unit (150) is configured to reconstruct a deformation of the transparent gel (130) during a pressing action from the event data, to determine the physical parameters of the pressing action from the reconstructed deformation, and to generate the music data according to the determined physical parameters.[9] The music production device (100) according to any one of [1] to [7], wherein the image sensor (140) comprises a plurality of pixels (141) and is capable to generate image frames indicating the intensity of light received at each pixel (141); the digital data are the image frames; and the control unit (150) is configured to reconstruct a deformation of the transparent gel (130) during a pressing action from the image frames, to determine the physical parameters of the pressing action from the reconstructed deformation, and to generate the music data according to the determined physical parameters.

[0010] The music production device (100) according to any one of [1] to [9], wherein the surface of the transparent gel (130) that receives the pressing action is curved, preferably with a deviation from a flat surface of more than 1 cm.

[0011] The music production device (100) according to any one of [1] to

[0010] , further comprising a sound reproduction unit (160) comprising loudspeakers; whereinthe sound reproduction unit (160) is configured to reproduce the music data generated by the control unit (150).

[0012] A method for operating the music production device (100) according to any of [1] to

[0011] , the method comprising receiving as user input a pressing action of a user on the transparent gel (130) that deforms the transparent gel (130) such that positions of the markers (135) change; with the image sensor (140), capturing image data of the change of positions of the markers (135), which image data allow deduction of physical parameters of the pressing action; converting the pressing action into digital signals by generating the digital signals from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action; and receiving the digital signals at the control unit (150) and generating by the control unit (150) music data according to the physical parameters of the pressing action represented by the received digital signals.

Claims

Claims1. A music production device comprising a user interface that is configured to receive user inputs from a user and to convert the user inputs into digital signals; and a control unit that is configured to receive the digital signals and to generate music data based on the received digital signals; wherein the user interface comprises at least one vision based tactile sensor that comprises a transparent gel on or in which markers are placed and an image sensor that observes the markers through the transparent gel; user inputs are received by receiving a pressing action of the user on the transparent gel that deforms the gel such that positions of the markers change; during the pressing action the image sensor is configured to capture image data of the change of positions of the markers, which image data allow deduction of physical parameters of the pressing action; the digital signals are generated from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action; and the music data are generated according to the physical parameters of the pressing action represented by the digital signals.

2. The music production device according to claim 1, wherein the physical parameters represented by the digital signals comprise at least a position of the pressing action on the transparent gel, a three-dimensional force vector generated by the pressing action, a pressure and / or a pressure distribution on the transparent gel caused by the pressing action, a shear force on the transparent gel caused by the pressing action, a size and / or shape of a contact area at which the vision based tactile sensor is contacted during the pressing action, and a temporal development of any of said parameters.

3. The music production device according to claim 1, wherein the user interface comprises one vision based tactile sensor that is virtually divided into a plurality of input regions; and / or the user interface comprises a plurality of vision based tactile sensors that constitute a plurality of input regions; and the control unit is configured to determine from the received digital signals in which of the input regions a pressing action has been received and to generate for pressing actions received at different input regions different music data that cause, when reproduced, different audible perceptions for a typical human.

4. The music production device according to claim 3, further comprising a memory for storing a plurality of samples as original music data; wherein the control unit is configured to generate the different music data from different original music data such that each piece of the different music data causes, when reproduced, an audible perception that is based on the respective sample.

5. The music production device according to claim 4, whereinat least some of the samples represent different tones at different pitches, played by one or several instruments and / or sang by one or a plurality of humans; and / or at least some of the samples represent different parts of a piece of music; and / or at least some of the samples represent different sounds.

6. The music production device according to claim 4, wherein the control unit is configured to determine the physical parameters of pressing actions and to vary the original music data based on the determined physical parameters.

7. The music production device according to claim 6, wherein based on the length of the pressing action a duration of the audible perception represented by the original music data is changed by increasing the duration for longer pressing action; and / or based on the strength of a pressure of the pressing action a volume of the audible perception represented by the original music data is changed, preferably by increasing the volume for larger pressures; and / or based on a movement of a position of the pressing action a pitch and / or a timbre of the audible perception represented by the original music data is changed, preferably by raising the pitch for a movement in one predetermined direction and lowering the pitch for a movement in the opposite direction.

8. The music production device according to claim 1, wherein the image sensor is an event-based vision sensor that comprises a plurality of pixels and is capable to generate event data for each of said pixels, whenever an intensity detected by a pixel is changed by an amount that is larger than an event detection threshold; the digital data are the event data; and the control unit is configured to reconstmct a deformation of the transparent gel during a pressing action from the event data, to determine the physical parameters of the pressing action from the reconstructed deformation, and to generate the music data according to the determined physical parameters.

9. The music production device according to claim 1, wherein the image sensor comprises a plurality of pixels and is capable to generate image frames indicating the intensity of light received at each pixel; the digital data are the image frames; and the control unit is configured to reconstruct a deformation of the transparent gel during a pressing action from the image frames, to determine the physical parameters of the pressing action from the reconstructed deformation, and to generate the music data according to the determined physical parameters.

10. The music production device according to claim 1, wherein the surface of the transparent gel that receives the pressing action is curved, preferably with a deviation from a flat surface of more than 1 cm.

11. The music production device according to claim 1, further comprising a sound reproduction unit comprising loudspeakers; wherein the sound reproduction unit is configured to reproduce the music data generated by the control unit.

12. A method for operating the music production device according to claim 1, the method comprising receiving as user input a pressing action of a user on the transparent gel that deforms the transparent gel such that positions of the markers change; with the image sensor, capturing image data of the change of positions of the markers, which image data allow deduction of physical parameters of the pressing action; converting the pressing action into digital signals by generating the digital signals from the image data obtained during the pressing action such as to represent the physical parameters of the pressing action; and receiving the digital signals at the control unit and generating by the control unit music data according to the physical parameters of the pressing action represented by the received digital signals.

Images