Interactive movement voice engine

Abstract

A method for generating an audio output is described. An image input of interactive movements by a user is received, captured by an image sensor. The interactive movements are mapped to a sequence of audio element identifiers. The sequence of audio element identifiers is processed to generate a musical sequence by performing a music theory rule enforcement on the sequence of audio element identifiers. An audio output representing the musical sequence is generated.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent No. 17 / 556,178, filed December 20, 2021, entitled "INTERACTIVE MOTION Speech Engine," the disclosure of which is incorporated herein by reference in its entirety.

[0002] Composing and sharing songs or musical sequences is a common way for individuals to build social bonds. Some people learn to play an instrument, such as the piano, guitar, or percussion, to compose for themselves and share with others. However, many songs or musical sequences may use several different instruments, while mastering one instrument requires years of practice and learning. To create a short musical sequence that others can comfortably listen to (i.e., has a high level of musicality), it may take several hours to write out the notes, play the separate instruments and record them on different tracks, and edit the tracks together. Also, the rules of music theory that promote musicality may be subtle and difficult to follow for users who have not studied music.

[0003] The embodiments are described with respect to these and other general considerations, and while relatively specific problems are discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the Background. Summary of the Invention [Problem to be solved by the invention]

[0004] Aspects of the present disclosure relate to generating audio output. [Means for solving the problem]

[0005] In one aspect, a method for generating an audio output is provided. An image input of interactive movements by a user is received, captured by an image sensor. The interactive movements are mapped to a sequence of audio element identifiers. The sequence of audio element identifiers is processed to generate a musical sequence by performing a music theory rule enforcement on the sequence of audio element identifiers. An audio output representing the musical sequence is generated.

[0006] In another aspect, a system for generating an audio output is provided, the system comprising one or more hardware processors configured with machine readable instructions to: receive image input of interactive movements by a user captured by an image sensor, map the interactive movements to a sequence of audio element identifiers, process the sequence of audio element identifiers to generate a musical sequence by performing a music theory rule enforcement on the sequence of audio element identifiers, and generate an audio output representative of the musical sequence.

[0007] In yet another aspect, a non-transitory computer-readable storage medium is provided that includes instructions executable by one or more processors that, when executed by the one or more processors, cause the one or more processors to receive image input of interactive movements by a user captured by an image sensor, map the interactive movements to a sequence of audio element identifiers, process the sequence of audio element identifiers to generate a musical sequence by performing a music theory rule enforcement on the sequence of audio element identifiers, and generate an audio output representative of the musical sequence.

[0008] This Summary is provided to introduce in a simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. [Brief description of the drawings]

[0009] Non-limiting and non-exhaustive examples are described with reference to the following drawings:

[0010] [Figure 1] FIG. 1 is a block diagram of an example of a system for generating audio output according to an example of the present disclosure.

[0011] [Diagram 2] FIG. 2 is a block diagram of an example of a user input processor of a computing device according to an example of the present disclosure.

[0012] [Diagram 3] FIG. 2 illustrates an exemplary output image for a graphical user interface according to an example of the present disclosure.

[0013] [Figure 4A] FIG. 2 illustrates an exemplary sequence of image inputs with facial expressions for generating audio output according to an example of the present disclosure.

[0014] [Figure 4B] FIG. 2 illustrates an exemplary image input with facial expression elements for generating an audio output, according to an example of the present disclosure.

[0015] [Diagram 5] 1 is a flowchart of an exemplary method for generating an audio output according to an example of the present disclosure.

[0016] [Figure 6]FIG. 1 is a block diagram illustrating example physical components of a computing device that can be used to implement aspects of the present disclosure.

[0017] [Figure 7] FIG. 1 is a simplified block diagram of a mobile computing device that can be used to implement aspects of the present disclosure. [Figure 8] FIG. 1 is a simplified block diagram of a mobile computing device that can be used to implement aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the disclosure. The embodiments may be embodied as a method, system, or apparatus. Thus, the embodiments may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the disclosure is limited by the appended claims and their equivalents.

[0019] As mentioned above, composing and sharing music is common, but composing often requires extensive practice to master technical skills on an instrument and music theory rules to improve musicality. The present disclosure provides methods and systems for music creation, such as generating audio output based on a user's interactive movements (e.g., moving hands, feet, arms, and / or legs). In some examples, a user may generate audio output by dancing in front of a camera of a computing device (e.g., a mobile phone), where the computing device maps the user's interactive movements to a sequence of audio element identifiers and then converts the sequence of audio element identifiers into a musical sequence and audio output. In other examples, a user may perform facial expressions (e.g., making a "happy" face) or facial expression elements (e.g., blinking, grinning, smiling) as interactive movements. By utilizing a user's interactive movements instead of keystrokes on a piano or fingering on a guitar, the challenge of mastering instrument skills is greatly reduced, making composition accessible to those who are not proficient on an instrument. In some examples, the computing device includes a music theory engine that generates musical sequences by performing music theory rule enforcement, e.g., by modifying sound element identifiers to improve their musicality. Thus, the music theory engine significantly reduces the demand for expertise in music theory, further increasing accessibility to music composition.

[0020] This and many other embodiments of computing devices are described herein. For example, FIG. 1 is a block diagram of an example of a system 100 for providing an interactive motion voice engine for generating audio output according to examples of the present disclosure. System 100 includes computing device 110 and computing device 120 communicatively coupled via network 150. Computing device 110 may be a smartphone, a mobile computer, or a mobile computing device (e.g., a Microsoft® Surface® device, a laptop computer, a notebook computer, an Apple® iPad®, etc.). TM Computing device 110 may be any type of computing device, including a tablet computer such as a mobile phone, a netbook, etc., or a fixed computing device such as a desktop computer or a PC (personal computer). In some examples, computing device 110 is a user device or client device of user 102, and computing device 120 is a server device. In some examples, computing device 120 is a network server, a cloud server, or other suitable distributed computing system. Computing device 120 may be operated and / or maintained by a social media platform, a cloud processing provider, a software-as-a-service provider, or other suitable entity. Computing device 110 and / or computing device 120 may be configured to run one or more software applications (or “applications”) and / or services that may be used by a user of computing device 110, and / or to manage hardware resources (e.g., processor, memory, etc.).

[0021] The computing device 110 includes an image sensor 112, a depth sensor 114, a user input processor 116, and a display 118. The image sensor 112 is configured to capture images and / or video of the user 102, for example, as the user 102 performs interactive movements. The image sensor 112 may be, for example, a front "selfie" camera or a rear camera of a smartphone. In various examples, images or still frames of the video may be used to identify facial expression elements, gestures, and user element movements performed by the user 102, as described below. The depth sensor 114 is configured to estimate a distance between the computing device 110 and the user 102, for example, to the hands, arms, legs, and / or head of the user 102. The depth sensor 114 may provide depth information that enhances images captured by the image sensor 112, thereby enabling estimation of a three-dimensional position of the user 102.

[0022] The user input processor 116 is configured to identify interactive movements of the user 102 based on the images captured by the image sensor 112 and / or the depth information from the depth sensor 114. Additionally, the user input processor 116 determines a user input identifier corresponding to the interactive movement. Advantageously, the user 102 does not need to use a touch screen, mouse, keyboard, or other physical input device to provide user input.

[0023] Typically, the user input identifier is a discrete identifier, such as an integer or other suitable value, that uniquely identifies an interactive movement already performed by the user 102. In various examples, the user input processor 116 may include one or more of a gesture processor that identifies gestures performed by the user 102, a facial expression processor that identifies facial expressions performed by the user 102, and / or a finger position processor that identifies finger positions of the user 102. Further details of the user input processor 116 are provided below with reference to FIG.

[0024] The display 118 is configured to present a user interface of the computing device 110. In various examples, the display 118 is a touch screen display of a smartphone, a monitor of a desktop computer, etc. The display 118 displays the images captured by the image sensor 112. The input image may be configured to display an output image including a graphical user interface overlaid on the input image, for example, the output image may provide the user 102 with real-time feedback about their interactive movements.

[0025] The computing device 120 includes an audio element processor 122, a music theory engine 124, a synthesizer 126, an effects engine 128, and a beat quantizer 130. The audio element processor 122 is configured to map user input identifiers representing interactive movements of a user to a sequence of audio element identifiers. The audio element identifiers are data structures that represent musical notes, samples, loops, and / or timbres for musical instruments (e.g., piano, acoustic guitar, trumpet). In some examples, the audio element identifiers are in Musical Instrument Digital Interface (MIDI) format. For example, the audio element identifiers include information about pitch, speed, vibrato, panning, clock signals for timing, etc. For example, a sound element identifier for middle C on a piano (fundamental frequency approximately 261.63 Hz using the A440 pitch standard) may have an integer value pitch that is 60 with a value range from 0 to 127 (e.g., two octaves below middle C has an integer value of 36, and D above middle C has an integer value of 62). In other examples, different identifiers are used for pitch (e.g., absolute or relative pitch between notes), duration of notes, volume, etc.

[0026] In some examples, the audio element processor 122 maps a single user input identifier to an audio element identifier for a note. In other words, a single interactive movement (e.g., a nod) is mapped to a single note with a start time and a stop time. In other examples, the audio element processor 122 maps a first user input identifier to an audio element identifier for a note start (i.e., a MIDI note-on event) and a second user input identifier to an audio element identifier for a note stop (i.e., a MIDI note-off event). In some examples, the same user input identifier is alternately mapped to a note-on event and a note-off event. In other examples, a subsequent different user input identifier is mapped to both a note-off of the previous note and a note-on of the current note.

[0027] Mapping a user input identifier to a sequence of audio element identifiers may include generating a single audio element identifier for a single user input identifier, or generating multiple audio element identifiers for a single user input identifier. For example, user 102 may hold one finger over an icon in a graphical user interface for a "one shot" of an audio sample, hold two fingers over an icon for a loop of the audio sample, etc.

[0028] In some examples, mapping the interactive movements includes selecting a set of predefined instruments from a plurality of sets of instruments and mapping the interactive movements to instruments within the selected set of predefined instruments. For example, the audio elements processor 122 may generate the plurality of sets of instruments using a neural network engine that identifies a set of predefined instruments from existing music samples (e.g., popular published music). In one example, each set of predefined groups of instruments is selected such that the instruments sound pleasing together. The sample sets may include percussion snare drum, kick drum, hi-hat, bass guitar, and overdriven electric guitar for a blues style set of instruments, violin, viola, and cello for a chamber quartet style set of instruments, drum machine, keyboard, and drum voice samples for a bass style set of instruments, and the like. The audio elements processor 122 may include other sets of instruments for different musical styles, such as country music, electronic music, hip hop music, jazz music, Latin music, pop music, rock music, and metal music. Conversely, less ideal sets of instruments may include banjo, slide whistle, distorted electric guitar, and drum set.

[0029] The music theory engine 124 is configured to perform music theory rule enforcement on the sequence of audio element identifiers to generate a musical sequence. A musical sequence is also a sequence of audio element identifiers, but more likely to have a high level of musicality. In general, the music theory engine 124 enforces rules that improve the musicality of the audio output 104 with respect to audio output that is strictly based on the sequence of audio element identifiers. For example, the music theory engine 124 enforces rules to reduce dissonance, change "bad" or "incorrect" notes to "good" notes (i.e., change notes that are not in the current chord to be in the current chord and change notes that are not in the current key signature to be in the current key signature), omit incorrect notes, or insert additional notes. The music theory engine 124 may enforce the rules by modifying at least one audio element identifier of the sequence of audio element identifiers that violates a music theory rule. Exemplary modifications include changing the pitch associated with at least one sound element identifier (e.g., to match a chord progression, scale, musical mode), omitting a sound element identifier (e.g., erasing "bad" notes), changing the duration of a sound element identifier, or changing other characteristics associated with the sound element identifier.

[0030] The music theory engine 124 may include one or more selectable rules that enforce various elements of music theory, such as maintaining the consistency of notes in a melody with a key signature, maintaining the harmony of notes in a chord, preserving notes in a chord progression, ensuring that chords with dissonant intervals are followed by chords with consonant intervals, etc.

[0031] The synthesizer 126 is configured to convert the musical sequences into audio output using synthesized or sampled audio and may be implemented, for example, by a library of MIDI output routines on the computing device 120 as part of a sound card driver. In another example, the synthesizer 126 is a keyboard or other suitable MIDI device that converts the musical sequences into audio output. The audio output may be provided as a WAV file, an MP3 file, or other suitable audio output file.

[0032] The effects engine 128 is configured to add musical effects, such as reverb, chorus, delay, overdrive, distortion, filter cutoff, envelope filter, flange, tremolo, or other suitable effects, to the audio output. In some examples, the effects engine 128 adds a drum backing track, loops, or other sampled sounds to the audio output.

[0033] The beat quantizer 130 is configured to modify the temporal characteristics (i.e., start and stop times) of the individual audio element identifiers of the musical sequence. In general, the beat quantizer 130 is configured to align the individual audio element identifiers to a time signature, a predefined rhythmic pattern, or other suitable temporal characteristics before the synthesizer 126 generates the audio output. For example, the beat quantizer 130 may align all audio element identifiers of the musical sequence to eighth notes in a 4 / 4 time signature. In some examples, the beat quantizer 130 randomly or pseudo-randomly selects a predefined rhythmic pattern from a plurality of predefined rhythmic patterns.

[0034] In some examples, one or more components of the computing device 110 may be omitted or moved to other devices. In one example, the depth sensor 114 is omitted from the computing device 110. In another example, the user input processor 116 is located in the computing device 120 and omitted from the computing device 110. In this example, the computing device 120 receives image input from the computing device 110. In yet another example, the effects engine 128 is omitted from the computing device 120. In another example, the audio element processor 122, the music theory engine 124, the synthesizer 126, the effects engine 128, and the beat quantizer 130 are located in the computing device 110 and omitted from the computing device 120. In this example, the computing device 120 and the network 150 are omitted and the processing for generating audio output may be performed by the computing device 110.

[0035] 2 is a block diagram of an example of a user input processor 200 of a computing device according to examples of the present disclosure. In some examples, the user input processor 200 generally corresponds to the user input processor 116 and generates a user input identifier as a discrete representation of an interactive movement from the user 102. The user input processor 200 includes a gesture processor 210, a facial expression processor 220, and a finger position processor 230.

[0036] The gesture processor 210 is configured to identify gestures performed by the user 102 and generate a corresponding user input identifier. Exemplary gestures may include shrug gestures, pointing gestures, nodding, waving, clapping, waving arms, lifting a foot, jumping, raising arms to a predefined position or angle, or other suitable gestures, and may be mapped to integer identifiers (e.g., 0 for shrug, 1 for point, 2 for nod, ...). Similarly, the facial expression processor 220 is configured to identify facial expressions performed by the user 102 and generate a corresponding user input identifier. Exemplary facial expressions may include a smiling face, a frowning face, a surprised face, an angry face, or other suitable facial expressions. In some examples, the user input processor 200 identifies changes in the user 102 over time, such as when the user's 102 face moves in a circular motion or rotates (i.e., when the user 102 turns around). In some examples, the gesture processor 210 identifies only a complete facial expression for a single user input identifier. In other examples, the gesture processor 210 identifies partial facial expressions as facial expression elements corresponding to different user input identifiers, such as one eye (or both eyes) open, one eye (or both eyes) closed, winking with one eye, blinking, open or closed mouth, grinning with part of the mouth, etc.

[0037] The finger position processor 230 is configured to identify a current position of one or more fingers of the user 102. If the current position of the finger overlaps a portion of the graphical user interface, the finger position processor 230 may provide a user input identifier that corresponds to the overlapping portion of the graphical user interface. In other examples, the user input processor 200 also identifies the movement of an object that overlaps the portion of the graphical user interface, such as a waving drumstick (or other implement that can mimic a drumstick, such as a pencil or pen). In some examples, the finger position processor 230 maps the finger position to a slider input or a pitch wheel. It pings the device and generates a corresponding user interface identifier.

[0038] In some examples, the user input processor 200 is configured to identify interactive movements that mimic the use of musical instruments such as "air guitar" or "air drumming." In the case of air guitar, the user input processor 200 may identify the strumming hand of the user 102 to select the start and stop of notes and another hand fretting the notes or chords to select the pitch of the virtual guitar. Additionally, the user input processor 200 may recognize the amount of mouth opening of the user 102 and map that amount to an effect, for example, a "wah" effect, where an open mouth corresponds to a pedal forward position of the wah pedal and a closed mouth corresponds to a pedal backward position of the wah pedal. In other examples, the user input processor 200 maps an open mouth to a different effect that matches an expression pedal, for example, volume, reverb, delay level, etc. In the case of air drumming, the user input processor 200 may map the lowest point of travel of the drumstick swing motion to the start of a musical note, and map the lateral and / or depth position to a pitch (e.g., a xylophone key or a cymbal pitch) or instrument (e.g., a hi-hat or snare drum).

[0039] Although the description herein is of a single user 102, the user input processor 200 may be configured to identify multiple users simultaneously. In some examples, the audio elements processor 122 may assign different users to different instruments within a given set of instruments (e.g., one user to acoustic guitar, one user to piano, one user to percussion).

[0040] In some examples, one or two of the processors 210, 220, and / or 230 may be omitted, for example, to simplify the operation of the user input processor 200 and / or to reduce power consumption of the computing device 110. In other examples, two or more of the processors 210, 220, and / or 230 may be combined with each other or with other elements of the system 100. In one such example, the processors 210, 220, 230 and the audio element processor 122, the music theory engine 124, the synthesizer 126, the effects engine 128, and the beat quantizer 130 are implemented as a single processor.

[0041] FIG. 3 illustrates an exemplary output image 300 for a graphical user interface 302 according to an example of the present disclosure. In some examples, the computing device 110 displays the output image 300 on the display 118. The output image 300 includes a graphical user interface 302 that is overlaid with the image input from the image sensor 112 and provides the user 102 with visual feedback about their interactive movements. The graphical user interface 302 includes a number of icons that represent buttons, triggers, switches, sliders, or other user interface elements with which the user 102 can virtually interact. As an example, the user 102 may move various user elements (e.g., fingers, hands, arms, feet, legs, head, or objects held in hand) to overlap with the icons of the graphical user interface 302. In the example illustrated in FIG. 3, the user 102 has a right hand 310 and a left hand 312, and the user 102 uses the right hand 310 and the left hand 312 to perform interactive movements.

[0042] The graphical user interface 302 includes icons 320, 322, 324, 326, 330, 332, 334, and 336 as buttons, and icons 340 and 342 as slider elements. In other examples, icons such as dials, momentary switches, latching switches, or other suitable icons may be implemented within the graphical user interface 302.

[0043] 3, user 102 "presses" icon 332 with left hand 312. The icons of graphical user interface 302 may be mapped to respective predefined sound element identifiers, for example, different notes of an instrument, parts of a drum kit, etc. The icons of graphical user interface 302 may also be mapped to discrete values, for example, volume levels, effect levels, frequency levels, etc. For example, icon 340 may correspond to a volume slider having a range of discrete values ​​from 0 to 100, and the relative position of icon 342 within icon 340 may correspond to a discrete value (e.g., a volume level that is 40).

[0044] 4A illustrates an example sequence 400 of image inputs with facial expressions for generating audio output, according to an example of the present disclosure. The user input processor 200 may receive the sequence 400, identify various or portions of facial expressions (e.g., facial expression elements) in the sequence 400 (e.g., using the facial expression processor 220), and map the facial expressions and / or facial expression elements to different audio element identifiers.

[0045] 4A, the user 102 executes facial expressions including a neutral expression 410 (e.g., neutral eyes, neutral eyebrows, and closed mouth), a smiling expression 412, a neutral expression 414, an excited expression 416, and a neutral expression 418. In one example, the user input processor 200 maps the neutral expressions 410, 414, and 418 to a MIDI all notes off or "rest" identifier, maps the smiling expression 412 to an audio element identifier corresponding to a first tuba sound, and maps the excited expression 416 to a second tuba sound. In this example, the beat quantizer 130 may time the various expressions to eighth notes within a 2 / 4 time signature function, and the synthesizer 126 may generate an audio output that generally corresponds to a polka "oom-pah" rhythm played on the off-beats on a tuba.

[0046] 4B is a diagram illustrating an example image input 460 with facial expression elements for generating an audio output according to an example of the present disclosure. Facial expression elements 420, 422, and 424 correspond to the user 102 winking with the left eye, winking with the right eye, and blinking, respectively. Facial expression elements 430, 432, 434, and 436 correspond to a grinning upright, a grinning upleft, a grinning downright, and a grinning downleft, respectively. Facial expressions 440, 442, 444, 446, and 448 include various facial expression elements, such as raised eyebrows, frowning, grimacing, smiling, and open mouth. Facial expression elements 450 and 452 correspond to a tilt of the user 102's head to the left and to the right, respectively. Although this specification has described several facial expressions and facial expression elements that can be mapped to different speech element identifiers, in other examples, the user input processor 200 may be configured to recognize other combinations of facial expression elements, facial expressions, or other inputs.

[0047] FIG. 5 is a flow chart of an exemplary method 500 for generating an audio output according to an example of the present disclosure. Unless otherwise indicated, the technical processes illustrated in these figures are performed automatically. In any given embodiment, some steps of the process may be repeated and operated using different parameters or data. Steps in an embodiment may also be performed in an order different from the top-to-bottom order arranged in FIG. 5. Steps may be performed sequentially, partially overlapping, or fully parallel. Thus, the order in which steps of the method 500 are performed may differ from one implementation of the process to another implementation of the process. Steps may be omitted, combined, renamed, regrouped, performed on one or more machines, or deviate from the illustrated flow, so long as the process performed is operable and consistent with at least one claim. The steps of FIG. 5 may be performed by computing device 110 (e.g., via user input processor 116, display 118), computing device 120 (e.g., via music theory engine 124, synthesizer 126, effects engine 128, or beat quantizer 130), or other suitable computing device.

[0048] Method 500 begins with step 502, in which image input of an interactive movement by a user is received as captured by an image sensor. In some examples, the image input corresponds to an image captured by image sensor 112 from user 102, such as the images shown in Figures 3, 4A and / or 4B. For example, the image input may be received by user input processor 116 or user input processor 200.

[0049] In step 504, the interactive movements are mapped to a sequence of audio element identifiers. In some examples, the audio element processor 122 maps the interactive movements to a sequence of audio element identifiers, as described above. For example, the audio element processor 122 is configured to map a user input identifier representing a user interactive movement to a sequence of audio element identifiers. The audio element identifiers are data structures that represent musical notes, samples, loops, and / or timbres for musical instruments (e.g., piano, acoustic guitar, trumpet). In some examples, the audio element identifiers are in MIDI format. For example, the audio element identifiers include information about pitch, speed, vibrato, panning, clock signals for timing, etc.

[0050] In some examples, the audio element processor 122 maps a single user input identifier to an audio element identifier for a note. In other words, a single interactive movement (e.g., a nod) is mapped to a single note with a start time and a stop time. In other examples, the audio element processor 122 maps a first user input identifier to an audio element identifier for a note start (i.e., a MIDI note-on event) and maps a second user input identifier to an audio element identifier for a note stop (i.e., a MIDI note-off event). In some examples, the same user input identifier is alternately mapped to a note-on event and a note-off event. In other examples, a subsequent different user input identifier is mapped to both a note-off of the previous note and a note-on of the current note. In some examples, mapping the interactive movement includes selecting a set of predefined instruments from a plurality of sets of instruments and mapping the interactive movement to an instrument within the selected set of predefined instruments.

[0051] In step 506, the sequence of voice element identifiers is processed to generate a musical sequence by performing music theory rule enforcement on the sequence of voice element identifiers. In some examples, as described above, the music theory engine 124 processes the sequence of voice element identifiers from the voice element processor 122 to generate a musical sequence. By enforcing the rules, the music theory engine 124 may reduce dissonance, change "bad" or "incorrect" notes to "good" notes (i.e., change notes that are not in the current chord to be in the current chord and change notes that are not in the current key signature to be in the current key signature), omit incorrect notes, or insert additional notes. The music theory engine 124 may enforce the rules by modifying at least one voice element identifier in the sequence of voice element identifiers that violates a music theory rule. Exemplary modifications include changing the pitch associated with at least one audio element identifier (e.g., to match a chord progression, scale, musical mode), omitting the audio element identifier (e.g., erasing a "bad" note), changing the duration of the audio element identifier, or changing other characteristics associated with the audio element identifier. The music theory engine 124 may include one or more selectable rules that enforce various elements of music theory, such as maintaining the consistency of notes in a melody with a key signature, maintaining the harmony of notes in a chord, preserving notes in a chord progression, ensuring that chords with dissonant intervals are followed by chords with consonant intervals, etc.

[0052] At step 508, an audio output representing the musical sequence is generated. In some examples, the synthesizer 126 generates the audio output based on the musical sequence provided by the music theory engine 124. In some examples, generating the musical sequence further includes adding effects to the audio output (e.g., by the effects engine 128) and / or aligning the audio element identifiers to a time signature, rhythm, and / or chord progression (e.g., by the beat quantizer 130).

[0053] Figures 6, 7, and 8 and the associated description provide an illustration of various operating environments in which various aspects of the disclosure may be implemented. However, the devices and systems shown and described with reference to Figures 6, 7, and 8 are for purposes of example and illustration, and are not intended to limit the numerous computing device configurations that may be used to implement various aspects of the disclosure as described herein.

[0054] 6 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600 that can be used to implement aspects of the present disclosure. The computing device components described below may have computer-executable instructions for implementing an audio output generating application 620 on a computing device (e.g., computing device 110, computing device 120), including computer-executable instructions for an audio output generating application 620 that can be executed to implement the methods disclosed herein. In a basic configuration, the computing device 600 may include at least one processing unit 602 and a system memory 604. Depending on the configuration and type of computing device, the system memory 604 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memory. The system memory 604 may include an operating system 605 and one or more program modules 606 suitable for executing an audio output generation application 620, such as one or more components related to Figures 1 and 2, in particular a user input processor 621 (e.g., corresponding to user input processor 116 or user input processor 200), an audio element processor 622 (e.g., corresponding to audio element processor 122), a music theory engine 623 (e.g., corresponding to music theory engine 124), a synthesizer 624 (e.g., corresponding to synthesizer 126), an effects processor 625 (e.g., corresponding to effects engine 128), and a beat quantizer 626 (e.g., corresponding to beat quantizer 130).

[0055] For example, an operating system 605 may be adapted to control the operation of computing device 600. Additionally, embodiments of the present disclosure may be implemented in conjunction with a graphics library, other operating systems, or any other application program, but are not limited to any particular application or system. This basic configuration is illustrated in FIG. 6 by those components within dashed line 608. Computing device 600 may have additional features or functionality. For example, computing device 600 may further include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 6 by removable storage device 609 and non-removable storage device 610.

[0056] As mentioned above, a number of program modules and data files may be stored in the system memory 604. The program modules 606 (e.g., audio output generation application 620), when executed on the processing unit 602, may perform processes including, but not limited to, aspects described herein. Other program modules usable in accordance with aspects of the present disclosure, particularly for generating audio output, may include a user input processor 621, an audio element processor 622, a music theory engine 623, a synthesizer 624, an effects processor 625, and a beat quantizer 626.

[0057] Additionally, embodiments of the present disclosure may be implemented in electrical circuits including discrete electronic components, packaged or integrated electronic chips including logic gates, circuits utilizing a microprocessor, or a single chip including electronic components or a microprocessor. For example, embodiments of the present disclosure may be implemented via a system-on-chip (SOC) in which each or multiple components shown in FIG. 6 may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or "baked") onto a chip substrate as a single integrated circuit. When operating via a SOC, the functionality described herein regarding the ability of a client to switch protocols may operate via application specific logic integrated with other components of the computing device 700 on a single integrated circuit (chip). Embodiments of the present disclosure may also be implemented using other technologies, including but not limited to mechanical, optical, fluidic, and quantum technologies, capable of performing logical operations such as AND, OR, and NOT. Additionally, embodiments of the present disclosure may be implemented within a general purpose computer or within any other circuit or system.

[0058] Computing device 600 may also have one or more input devices 612, such as a keyboard, mouse, pen, acoustic or voice input device, touch or swipe input device, etc. Output devices 614, such as a display, speakers, printer, etc. may also be included. The above devices are examples and other devices may be used. Computing device 600 may include one or more communications connections 616 that enable communication with other computing devices 650. Examples of suitable communications connections 616 include, but are not limited to, radio frequency (RF) transmitter, receiver and / or transceiver circuitry, a universal serial bus (USB), a parallel port, and / or a serial port.

[0059] The term computer readable media as used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, Electrically Erasable Read-Only Memory (EEPROM), Flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article used to store information and accessible by computing device 600. Any such computer storage media may be part of computing device 600. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0060] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may refer to a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared and other wireless media.

[0061] 7 and 8 show a mobile computing device 700, such as a mobile phone, a smartphone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., that can be used to implement embodiments of the present disclosure. In some aspects, the client may be a mobile computing device. Referring to FIG. 7, one aspect of a mobile computing device 700 for implementing these aspects is shown. In a basic configuration, the mobile computing device 700 is a handheld computer having both input and output elements. The mobile computing device 700 typically includes a display 705 and one or more input buttons 710 that allow a user to input information into the mobile computing device 700. The display 705 of the mobile computing device 700 can also function as an input device (e.g., a touch screen display). An optional secondary input element 715 (if included) further allows for another user input. The secondary input element 715 may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 700 may incorporate more or fewer input elements. For example, in some embodiments, the display 705 may not be a touch screen. In yet another alternative embodiment, the mobile computing device 700 is a mobile telephone system, such as a cellular telephone. The mobile computing device 700 may include a front camera 730. The mobile computing device 700 may further include an optional keypad 735. The optional keypad 735 may be a physical keypad or a "soft" keypad generated on a touch screen display. In various embodiments, the output elements include the display 705 for displaying a graphical user interface (GUI), a visual indicator 720 (e.g., a light emitting diode), and / or an audio transducer 725 (e.g., a speaker).In some embodiments, the mobile computing device 700 incorporates a vibration transducer for providing haptic feedback to the user. In yet other embodiments, the mobile computing device 700 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port) to send and receive signals to and from external devices.

[0062] Figure 8 7 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, the mobile computing device 700 can implement several aspects by incorporating a system (e.g., architecture) 802. In one embodiment, the system 802 is implemented as a "smartphone" capable of executing one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, games, and media client / player). In some aspects, the system 802 is integrated as a computing device such as an integrated personal digital assistant (PDA), wireless phone, etc. The system 802 may include a display 805 (similar to the display 705), such as a touch screen display, or other suitable user interface. The system 802 may also include an optional keypad 835 (similar to the keypad 735) and one or more peripheral ports 830, such as input and / or output ports for audio, video, control signals, or other suitable signals.

[0063] In some examples, the system 802 may include a processor 860 coupled to the memory 862. The system 802 may also include a special purpose processor 861, such as a neural network processor. One or more application programs 866 may be loaded into the memory 862 and run on or in conjunction with the operating system 864. Examples of application programs include a telephone dialing program, an email program, a personal information manager (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 802 also includes a non-volatile storage area 868 in the memory 862. The non-volatile storage area 868 may be used to store persistent information that should not be lost when the system 802 is powered off. The applications 866 may use and store information in the non-volatile storage area 868, such as emails or other messages used by an email application, in the non-volatile storage area 868. A synchronization application (not shown) also resides on the system 802 and is programmed to interact with a corresponding synchronization application resident on the host computer to maintain synchronization between the information stored in the non-volatile storage area 868 and corresponding information stored on the host computer.

[0064] The system 802 includes a power supply 870, which may be implemented as one or more batteries. The power supply 870 may also include an external power source, such as a powered storage base or an AC adapter, for replenishing or recharging the batteries.

[0065] System 802 may further include a wireless interface layer 872 that performs functions to transmit and receive radio frequency communications. Wireless interface layer 872 facilitates wireless connectivity between system 802 and the "outside world" via a communications carrier or service provider. Transmissions to and from wireless interface layer 872 are performed under the control of operating system 864. In other words, communications received by wireless interface layer 872 may be delivered to application programs 866 via operating system 864, and vice versa.

[0066] The visual indicator 820 can be used to provide a visual notification and / or the audio interface 874 can be used to generate an audible notification via the audio transducer 725 (e.g., audio transducer 725 shown in FIG. 7). In the illustrated embodiment, the visual indicator 820 is a light emitting diode (LED) and the audio transducer 725 can be a speaker. These devices can be directly coupled to the power source 870 so that when activated, they remain on for a duration specified by the notification mechanism even if the processor 860 and other components are turned off to conserve battery power. The LED can be programmed to remain illuminated indefinitely until the user takes an action that indicates the powered on state of the device. The audio interface 874 is used to provide audible signals to the user and receive audible signals from the user. For example, the audio interface 874, in addition to being coupled to the audio transducer 725, can be coupled to a microphone to receive audible input, for example, to facilitate a telephone conversation. According to an embodiment of the present disclosure, the microphone can also function as an audio sensor to facilitate control of notifications, as described below. The system 802 may further include a video interface 876 that enables operation of peripherals 830 (eg, an on-board camera) to record still images, video streams, and the like.

[0067] A mobile computing device 700 implementing system 802 may have additional features or functionality. For example, mobile computing device 700 may further include additional data storage devices (removable and / or non-removable) such as magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 8 by non-volatile storage 868.

[0068] As described above, data / information generated or captured by the mobile computing device 700 and stored via the system 802 may be stored locally on the mobile computing device 700, or the data may be stored in any number of storage media accessible to the device via the wireless interface layer 872, or via a wired connection between the mobile computing device 700 and another computing device associated with the mobile computing device 700 (e.g., a server computer in a distributed computing network, such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 700, via the wireless interface layer 872, or via a distributed computing network. Similarly, such data / information may be readily transmitted between computing devices for storage and use in accordance with known data / information transmission and storage means, including electronic mail and collaborative data / information sharing systems.

[0069] It should be noted that Figures 7 and 8 are provided to illustrate the present method and system and are not intended to limit the present disclosure to any particular sequence of steps or combination of hardware or software components.

[0070] The terms "at least one," "one or more," "or," and "and / or" are open-ended expressions that are conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," "A, B, and / or C," and "A, B, or C" means A only, B only, C only, A and B, A and C, B and C, or A, B and C.

[0071] The term "an" or "an" entity means one or more of that entity. Thus, the terms "one," "one or more," and "at least one" may be used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" may be used interchangeably.

[0072] As used herein, the term "automatic" and variations thereof refer to any process or operation, usually continuous or semi-continuous, that is performed without significant manual input as the process or operation is performed. However, performance of a process or operation can be performed automatically with significant or insignificant manual input, provided that the input is accepted prior to the performance of the process or operation. A manual input is considered significant input if it affects how the process or operation is performed. A manual input to consent to the performance of a process or operation is not considered "substantial."

[0073] Any of the steps, functions, and operations discussed herein may be performed continuously and automatically.

[0074] The exemplary systems and methods of the present disclosure have been described in conjunction with a computing device. However, in order to avoid unnecessarily obscuring the present disclosure, the foregoing description has omitted some known structures and devices. This omission should not be construed as limiting. Specific details are described to provide an understanding of the present disclosure. However, it should be understood that the present disclosure may be implemented in a variety of ways in addition to the specific details set forth herein.

[0075] Furthermore, although in the exemplary embodiments illustrated herein, various components of the system are shown to be co-located, some components of the system may be located remotely in a distal portion of a distributed network, such as a LAN and / or the Internet, or may be located in a dedicated system. Thus, it should be understood that the components of the system may be coupled to one or more devices, e.g., a server, a communication device, or may be co-located on a particular node of a distributed network, such as an analog and / or digital telecommunications network, a packet-switched network, or a circuit-switched network. As will be appreciated from the foregoing description, for reasons of computing efficiency, the components of the system may be located anywhere within the distributed network of components without affecting the operation of the system.

[0076] It should further be understood that the various links connecting the elements may be wired or wireless links, or any combination thereof, or any other known or future developed elements capable of providing data to and / or communicating data from the connected elements. These wired or wireless links may also be secure links and may be capable of communicating encrypted information. For example, the transmission medium used as the link may be any suitable carrier of electrical signals, including coaxial cable, copper wire, and optical fiber, or may take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0077] Although the flowcharts have been discussed and illustrated with reference to particular sequences of events, it should be understood that modifications, additions, and omissions may be made to the sequences without substantially affecting the operation of the disclosed configurations and aspects.

[0078] Several variations and modifications of the disclosure may be used, and it is possible to provide some features of the disclosure and not other features.

[0079] In other settings, the systems and methods of the present disclosure may be implemented in combination with special purpose computers, programmed microprocessors or microcontrollers and peripheral integrated circuit elements, ASICs or other integrated circuits, digital signal processors, hardwired electronic or logic circuits (e.g., discrete element circuits), programmable logic devices or gate arrays (e.g., PLDs, PLAs, FPGAs, PALs), special purpose computers, any similar devices, and the like. Overall, any device or means capable of implementing the methods set forth herein can be used to implement various aspects of the present disclosure. Exemplary hardware that can be used in the present disclosure include: The hardware includes computers, handheld devices, telephones (e.g., cellular telephones, Internet-enabled telephones, digital telephones, analog telephones, hybrid telephones, etc.), and other hardware known in the art. Some of these devices include a processor (e.g., single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. Alternative software implementations may also be constructed to implement the methods described herein, including, but not limited to, distributed or component / object distributed processing, parallel processing, or virtual machine processing.

[0080] In yet another configuration, the disclosed methods may be readily implemented in combination with software using object or object-oriented software development environments that provide portable source code usable on a variety of computer or workstation platforms. Alternatively, the disclosed systems may be implemented partially or fully in hardware using standard logic circuits or VLSI designs. Whether to use software or hardware to implement a system according to the present disclosure depends on the speed and / or efficiency requirements of the system, the particular functionality, and the particular software or hardware system or microprocessor or microcomputer system being used.

[0081] In yet another configuration, the disclosed methods may be implemented in part in software that may be stored on a storage medium and executed on a programmed general purpose computer, special purpose computer, microprocessor, etc. in cooperation with a controller and memory. In these instances, the disclosed systems and methods may be implemented as programs, e.g., applets, JAVA, or CGI scripts, installed on a personal computer, as resources resident on a server or computer workstation, as routines installed in dedicated measurement systems, system components, etc. The systems and / or methods may also be realized by physically incorporating them into software and / or hardware systems.

[0082] This disclosure is not limited to the standards and protocols described. Other similar standards and protocols not described herein already exist and are included in this disclosure. Furthermore, the standards and protocols described herein, and other similar standards and protocols not described herein, are periodically replaced by faster or more efficient equivalents having substantially the same functionality. Such replacement standards and protocols having the same functionality are considered to be equivalents included in this disclosure.

[0083] In various configurations and aspects, the present disclosure includes the components, methods, processes, systems, and / or devices as illustrated and described herein, including various combinations, subcombinations, and subsets thereof. Those skilled in the art, upon understanding the present disclosure, will understand how to make and use the systems and methods of the present disclosure. The present disclosure, in various configurations and aspects, includes providing devices and processes in the absence of items that may have been used in previous devices or processes to, for example, improve performance, provide ease of use, and / or reduce implementation costs, and in the absence of items not illustrated and / or described herein or in its various configurations or aspects.

[0084] The present disclosure relates to systems and methods for generating audio output, at least according to the examples provided in the following sections.

[0085] (A1) In one aspect, some examples include a method for generating an audio output, the method including receiving image input of interactive movements by a user captured by an image sensor, mapping the interactive movements to a sequence of audio element identifiers, processing the sequence of audio element identifiers to generate a musical sequence by performing a music theory rule enforcement on the sequence of audio element identifiers, and generating an audio output representative of the musical sequence.

[0086] (A2) In some examples of A1, processing the sound element identifiers includes modifying at least one sound element identifier of a sequence of sound element identifiers that violates a music theory rule, and generating the musical sequence based on the modified sound element identifier.

[0087] (A3) In some examples of A1 to A2, modifying the at least one voice element identifier includes changing a pitch associated with the voice element identifier.

[0088] (A4) In some examples of A1-A3, changing the pitches includes matching a chord progression that satisfies the music theory rules.

[0089] (A5) In some examples of A1-A4, modifying the at least one sound element identifier includes omitting the at least one sound element identifier when generating the musical sequence.

[0090] (A6) In some examples of A1 to A5, modifying the at least one voice element identifier includes changing a duration of the at least one voice element identifier.

[0091] (A7) In some examples of A1-A6, mapping the interactive movements includes selecting a set of predetermined instruments from a plurality of sets of instruments, and mapping the interactive movements to instruments within the selected set of predetermined instruments.

[0092] (A8) In some examples of A1-A7, the method further includes generating the plurality of instrument sets using a neural network engine that identifies a set of predefined instruments from a music sample.

[0093] (A9) In some examples of A1-A8, the method further includes displaying to the user an output image including a graphical user interface overlaid on the image input, and the interactive movement includes a user element of the user overlapping with the graphical user interface.

[0094] (A10) In some examples of A1 to A9, the user element of the user is the user's finger, hand, arm, foot, and / or leg.

[0095] (A11) In some examples of A1-A10, the graphical user interface includes a plurality of icons corresponding to a plurality of predefined sound element identifiers, and mapping the interactive movements includes mapping an interactive movement having a user element overlapping an icon to a predefined sound element identifier corresponding to the overlapped icon.

[0096] (A12) In some examples of A1 to A11, the plurality of predetermined voice element identifiers include a single element identifier and a multi-element identifier.

[0097] (A13) In some examples of A1 to A12, the interactive movement is a facial expression element performed by the user.

[0098] (A14) In some examples of A1 to A13, the interactive movement is a gesture performed by the user.

[0099] In yet another aspect, some examples include a computing system comprising one or more processors and a memory coupled to the one or more processors, the memory storing a plurality of instructions, the one or more instructions, when executed by the one or more processors, causing the one or more processors to perform any of the methods described herein (e.g., A1-A14 above).

[0100] In yet another aspect, some examples include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein (e.g., A1-A14 above).

[0101] In yet another aspect, some examples include a computing system comprising one or more processors and a memory coupled to the one or more processors, the memory storing a plurality of instructions, the one or more instructions, when executed by the one or more processors, causing the one or more processors to perform any of the methods described herein (e.g., method 500 above).

[0102] In yet another aspect, some examples include a non-transitory computer-readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein (e.g., method 500 above).

[0103] For example, aspects of the disclosure are described above with reference to block diagrams and / or operational descriptions of methods, systems, and computer program products according to aspects of the disclosure. The functions / acts noted in the blocks may occur in an order different from the order shown in any flowchart. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functions / acts involved.

[0104] The description and illustration of one or more aspects provided in this application are not intended to restrict or limit the scope of the claimed disclosure in any manner. The aspects, examples, and details described herein are deemed sufficient to convey ownership and to enable others to make and use the best mode of the claimed disclosure. The claimed disclosure should not be construed as being limited to the aspects, examples, or details described herein. Various features (structural features and method features), whether illustrated or described in combination or individually, are intended to be selectively included or omitted to form an embodiment having a particular set of features. The description and illustrations provided in this application may enable those skilled in the art to envision changes, modifications, and alternative embodiments that fall within the spirit of the broader aspects of the general inventive concept embodied in this application without departing from the broader scope of the claimed disclosure. [Explanation of symbols]

[0105] 104 Audio Output 110 Computing Equipment 112 Image Sensor 114 Depth Sensor 116 User Input Processor 118 Display 120 Computing Devices 122 Audio Element Processor 124 Music Theory Engine 126 Synthesizer 128 Effects Engine 130 Beat Quantizer 150 Network 200 User Input Processor 210 Gesture Processor 220 Facial Expression Processor 230 Finger Position Processor 502 receives an image input of an interactive movement by a user captured by an image sensor. 504 Mapping interactive actions to sequences of audio element identifiers 506 Process a sequence of audio element identifiers to generate a musical sequence by performing a music theory rule enforcement on the sequence of audio element identifiers. 508 Generate audio output representing a musical sequence 600 Computing Devices 602 Processing Unit 604 System Memory 605 Operating Systems 606 Program Module 609 Removable Storage Devices 610 Non-removable storage devices 612 Input Device 614 Output Device 616 Communication Connection 620 Applications 621 User Input Processor 622 Audio Element Processor 623 Music Theory Engine 624 Synthesizer 625 Effects Processor 626 Beat Quantizer 650 Other Computing Devices 861 Dedicated Processor 860 Processor 805 Display 830 Peripheral Port 835 Keypad 862 Memory 866 Apps 868 Storage device 870 Power supply 876 Video Interface 874 Voice Interface 872 Radio Interface Layer

Claims

1. 1. A method for generating an audio output, comprising: receiving image input of interactive movements by a user captured by an image sensor; mapping the interactive movements to a sequence of audio element identifiers; processing the sequence of audio element identifiers to generate a musical sequence by performing music theory rule enforcement on the sequence of audio element identifiers; and generating an audio output representing the musical sequence.

2. 2. The method of claim 1 , wherein processing the sound element identifiers comprises: modifying at least one sound element identifier of the sequence of sound element identifiers that violates a music theory rule; and generating the musical sequence based on the modified sound element identifier.

3. The method of claim 2 , wherein modifying the at least one voice element identifier comprises: changing a pitch associated with the voice element identifier.

4. The method of claim 3 , wherein modifying the pitches comprises matching a chord progression that satisfies the music theory rules.

5. The method of claim 2 , wherein modifying the at least one sound element identifier comprises omitting the at least one sound element identifier when generating the musical sequence.

6. The method of claim 2 , wherein modifying the at least one sound element identifier comprises changing a duration of the at least one sound element identifier.

7. 2. The method of claim 1, wherein mapping the interactive movements comprises: selecting a predetermined set of instruments from a plurality of sets of instruments; and mapping the interactive movements to instruments within the selected predetermined set of instruments.

8. 8. The method of claim 7, further comprising: generating the plurality of instrument sets using a neural network engine that identifies a set of predefined instruments from a musical sample.

9. 2. The method of claim 1, further comprising: displaying to the user an output image including a graphical user interface overlaid on the image input, the interactive movement including a user element of the user overlapping with the graphical user interface.

10. The method of claim 9 , wherein the user elements of the user are the user's fingers, hands, arms, feet, and / or legs.

11. 10. The method of claim 9, wherein the graphical user interface includes a plurality of icons corresponding to a plurality of predefined sound element identifiers, and mapping the interactive movements includes mapping an interactive movement having a user element overlapping an icon to a predefined sound element identifier corresponding to the overlapped icon.

12. The method of claim 11 , wherein the plurality of predefined audio element identifiers includes single element identifiers and multi-element identifiers.

13. The method of claim 1 , wherein the interactive movement is a facial expression performed by the user.

14. The method of claim 1 , wherein the interactive movement is a gesture performed by the user.

15. 1. A system for generating an audio output, the system comprising: one or more hardware processors configured with machine-readable instructions to perform the following: receive image input of interactive movements by a user captured by an image sensor; map the interactive movements to a sequence of audio element identifiers; process the sequence of audio element identifiers to generate a musical sequence by performing music theory rule enforcement on the sequence of audio element identifiers; and generate an audio output representing the musical sequence.

16. 16. The system of claim 15, wherein the one or more hardware processors are further configured with machine readable instructions to: modify at least one sound element identifier of the sequence of sound element identifiers that violates a music theory rule; and generate the musical sequence based on the modified sound element identifier.

17. 16. The system of claim 15, wherein the one or more hardware processors are further configured with machine readable instructions to: modify a pitch associated with the sound element identifier.

18. A non-transitory computer-readable storage medium containing instructions executable by one or more processors, which instructions, when executed by the one or more processors, cause the one or more processors to: receive image input of interactive movements by a user captured by an image sensor; map the interactive movements to a sequence of audio element identifiers; process the sequence of audio element identifiers to generate a musical sequence by performing a music theory rule implementation on the sequence of audio element identifiers; and generate an audio output representing the musical sequence.

19. 20. The non-transitory computer-readable storage medium of claim 18, wherein the instructions are executable by the one or more processors and, when executed by the one or more processors, cause the one or more processors to: modify at least one sound element identifier of the sequence of sound element identifiers that violates a music theory rule; and generate the musical sequence based on the modified sound element identifier.

20. 20. The non-transitory computer-readable storage medium of claim 18, wherein the instructions are executable by the one or more processors and, when executed by the one or more processors, cause the one or more processors to: modify a pitch associated with the sound element identifier.

Images