Motion-Based Media Creation

Abstract

A system for motion-based media creation includes an article of footwear or clothing having at least one accelerometer or inertial measurement unit operative to monitor spatial motion of at least a portion of the footwear or clothing and generate a data stream indicative of the monitored spatial motion. The system further includes a processor in wireless communication networked with the footwear or clothing. The processor is configured to receive the data stream from the article of footwear or clothing, identify at least one movement primitive from the received data stream, and trigger the playback of an audio sample or a visual effect in response to the identified at least one movement primitive.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority from U.S. Provisional Patent Application No. 62 / 939,309, filed November 22, 2019, and U.S. Provisional Patent Application No. 63 / 032,689, filed May 31, 2020, the entireties of both of which are incorporated by reference in their entireties.

[0002] The present disclosure relates to a system for converting real-time motion of a user sensed by an electronic article of footwear or apparel into an audio and / or visual composition. [Background technology]

[0003] In the music industry, it is common for composers or DJs to layer electronic tracks to create musical compositions. In creating these compositions, artists may rely on individual audio samples that can be triggered by a keyboard or other Musical Instrument Digital Interface (MIDI) controller. The resulting output may be creatively expressive and may require exceptional skill to arrange, but typically lacks performance attributes.

[0004] When considering physical performances (e.g., dance performances), often the artist may be responding or performing in sync with the musical piece, but the movements are typically merely a choreographed response to the music. Summary of the Invention

[0005] The present disclosure generally relates to a system capable of capturing a user's movements and translating the sensed movements into a form of creative expression. The system may include an article of footwear or clothing having at least one accelerometer or inertial measurement unit operative to monitor spatial motion of at least a portion of the footwear or clothing and generate a data stream indicative of the monitored spatial motion. The system further includes a processor in networked wireless communication with the footwear or clothing. The processor is configured to receive the data stream from the footwear or clothing; identify at least one motion primitive from the received data stream; and trigger playback of an audio sample or a visual effect in response to the identified at least one motion primitive. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram of one embodiment of a system for motion-based media creation.

[0007] [Diagram 2] 1 is a schematic diagram of an electronic motion-sensing article of footwear.

[0008] [Diagram 3] 1 is a schematic flow diagram of a method of operation for a system for motion-based media creation.

[0009] [Figure 4] FIG. 2 is a schematic functional block diagram of the operation of a system for motion-based media creation.

[0010] [Diagram 5]A schematic diagram of multiple wearables, each translating sensed movements into audio / visual output according to a common set of rules.

[0011] [Figure 6] A schematic diagram of multiple wearables grouped into pods, where each pod translates the movements of its associated wearable into audio / visual output via a pod-specific set of rules.

[0012] [Figure 7] FIG. 1 is a schematic diagram of multiple wearables, each converting sensed motion into audio / visual output by a distinct set of rules.

[0013] [Figure 8] FIG. 1 is a schematic diagram of a user using the system to perform a motion-based challenge.

[0014] [Figure 9] FIG. 1 is a schematic diagram of a user device displaying augmented reality visual effects resulting from sensed movements of a connected wearable.

[0015] [Figure 10] FIG. 1 is a schematic diagram of a flash mob where each participant has a wearable with location-based access to location-based audio / visual output.

[0016] [Figure 11] FIG. 1 is a schematic partial cross-sectional side view of a motion activated light projector within an article of footwear.

[0017] [Figure 12] 1 is a schematic perspective view of an article of footwear comprising a plurality of light emitting elements;

[0018] [Figure 13] FIG. 1 is a schematic rear view of an article of clothing comprising a plurality of light emitting elements.

[0019] [Figure 14] 1 is a schematic partial cross-sectional view of a vibration transducer disposed within an article of footwear;

[0020] [Figure 15] FIG. 1 is a schematic side view of an inductive charger used to charge multiple wearables.

[0021] [Figure 16] FIG. 16 is a schematic side view of the induction charger of FIG. 15 used as a party light.

[0022] [Figure 17] FIG. 1 is a schematic perspective view of an interactive retail kiosk.

[0023] [Figure 18] FIG. 18 is a schematic flow diagram of a method for obtaining a physical or electronic object or code using an interactive retail kiosk or the like of FIG. 17.

[0024] [Figure 19] FIG. 1 is a schematic diagram of a group of performers using connected wearables to affect a performance within a hosted virtual world.

[0025] [Figure 20] FIG. 1 is a schematic diagram of a wearable used to augment live video broadcasting. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The following description and accompanying figures disclose a system that uses directly sensed body movements to trigger the playback of one or more audio samples or visual effects. This technology is intended to create new forms of expression in which the movements of a dancer or athlete can drive the creation of an electronic multimedia experience.

[0027] In addition to simply providing tools to compose audio / visual (A / V) experiences, some embodiments of the present technology may enable social collaboration between multiple users. For example, in some configurations, multiple users of the system may work together, either locally or in a networked manner, to create joint / collaborative A / V compositions. In another example, multiple networked users may issue and / or respond to motion-based challenges to one another.

[0028] In a collaborative context, some embodiments of the technology may enable multiple members of a dance troupe or social network to collaborate in creating an A / V composition, much like a symphony plays out. In particular, each user or small group of users may have unique sound or visual effects associated with their movements. During the performance, the combined sound output resulting from each member's movements creates an A / V composition based on the performance, in which the bodies of the various users become "instruments."

[0029] Challenges in the context of social media have become increasingly popular for individuals to challenge each other online via various social media platforms. These challenges often involve users performing one or more actions or dances to a particular audio clip. One example of such a challenge involved a user videotaping themselves pouring ice water over their head and then issuing the same challenge to another user. Other challenges involve performing a particular or improvised dance sequence to a portion of a song. In each example, a user may videotape themselves performing the dance / action and post the resulting video clip to an online video hosting service. Examples of such hosting services include TIKTOK and DOUYIN, both operated by Beijing ByteDance Technology Co Ltd., or YOUTUBE, operated by Youtube, LLC, a subsidiary of Google, LLC. As described below, the technology may be suitable for similar "challenges."

[0030] In some embodiments, the output of the user's expressions may exist only in a separate medium and / or for consumption by others (i.e., "audiences") remote from the user. For example, the user's movements may be used to trigger one or more audio and / or visual effects in a virtual environment, such as may exist in a video game. Alternatively, these effects may be presented in an augmented reality (AR) environment, where they may be superimposed on the natural perception of the real world. In such an AR context, the effects may be delivered to user devices of those attending the event, such that they may be superimposed on the user's view of the real world, as in the case of AR display glasses. Alternatively, the effects may be superimposed on a captured video feed, e.g., a streamed video (Internet or television) broadcast, that may be viewed by a user device, such as a mobile phone or television.

[0031] As shown generally in FIG. 1, a motion-based media creation system 10 may generally include electronic motion-sensing articles of footwear or clothing 20 networked with a user console 30. For purposes of this disclosure, the "electronic motion-sensing articles of footwear or clothing 20" may generally be referred to as "wearables 20." Each wearable 20 may be fashioned in a style similar to a conventional article of footwear or clothing, but may have additional electronic functionality that enables the detection and external transmission of motion data. Examples of suitable articles of footwear may include shoes, boots, sandals, cleated footwear, and the like. Similarly, examples of suitable articles of apparel may include shirts, jackets, pants, shorts, socks, compression sleeves, gloves, hats, armbands / bracelets, and the like.

[0032] In the present technology as shown in Figure 2, the wearable 20 may include both motion sensing circuitry 22 and communications circuitry 24. The motion sensing circuitry 22 may include at least one accelerometer or inertial measurement unit (IMU) 26 that operates to sense spatial movement of at least a portion of the wearable 20 and generate a data output stream representative of the sensed movement. In some embodiments, the motion sensing circuitry 22 may further include a processor 28, memory, and / or any buffering or filtering circuitry that may be required to prepare the data output stream for transmission and / or to convert the raw data output into a streaming sequence of recognized motion primitives (described in more detail below).

[0033] The communication circuitry 24 coupled to the motion sensing circuitry 22 is configured to transmit a data stream externally to the user console 30. The communication circuitry 24 may include one or more transceivers, antennas, and / or memory required to facilitate real-time or near real-time data transmission. This data transmission may occur according to any suitable wireless standard, but particularly suitable communication protocols include any of the following: IEEE 802.11, 802.15, 1914.1, 1914.3; BLUETOOTH® or BLUETOOTH LOW ENERGY® (or other similar protocols / standards established by the Bluetooth SIG); 4G LTE Cellular, 5G, 5G NR; or similar wireless data communication protocols.

[0034] Referring again to FIG. 1, the user console 30 in communication with the wearable 20 may include computing device operating software or firmware specifically designed to cause the computing device to perform as described below. The user console may include a processor 32, a memory 34, and a user interface 36. In general, the processor 32 used in the present system 10 may be embodied as one or more digital computers, data processing devices, and / or digital signal processors (DSPs), which may have one or more microcontrollers or central processing units (CPUs), read only memory (ROM), random access memory (RAM), electrically erasable programmable read only memory (EEPROM), high speed clocks, analog-to-digital (A / D) circuits, digital-to-analog (D / A) circuits, input / output (I / O) circuits, and / or signal conditioning and buffering electronics. Examples of suitable user consoles include smartphones, tablet computers, laptop computers, desktop computers, and the like.

[0035] The user interface 36 may be configured to provide the user with the ability to see and / or hear the available A / V effects while allowing the user to build a correspondence table. In some embodiments, the user interface 36 may include one or more displays 38 operative to output visual information to the user and / or one or more user input devices 40 operative to receive input from the user. Examples of user input devices 40 include a touch screen / digitizer, a mouse, a keyboard, and / or a control panel having multiple rotary knobs and / or buttons, a camera, a gesture-based input device, an AR / VR virtual selection, etc.

[0036] 1 , in addition to communicating with the wearable 20, in some embodiments the user console 30 may also communicate with one or more video sources 42, one or more audio sources 44, one or more audio output devices 46, one or more visual effects controllers 48, and / or a distributed computing network 50 (e.g., the Internet). The video sources 42 may include a live video stream (e.g., from a digital camera), a previously recorded video stream, a digital memory storing one or more previously recorded videos, and / or the audio sources 44 may include one or more musical instruments, keyboards, synthesizers, data files, a collection of audio samples, and / or the like. In some embodiments, the video sources 42 and / or audio sources 44 may be local to the user console 30 or may be provided on a common local area network. However, in other embodiments, one or both of these sources 42, 44 may be remote from the user console 30 and / or may be hosted by a computer accessible only through the console's network connection.

[0037] The one or more audio output devices 46 may include one or more speakers, amplifiers, headphones, or other devices operable to broadcast an audible sound in response to a received digital or analog audio signal. Through these audio output devices 46, the user console 30 outputs one or more audio samples in response to sensed motion. The visual effects controller 48 may include one or more devices operable to illuminate one or more lights, initiate one or more lighting sequences, initiate one or more firework effects, etc. In some embodiments, the visual effects controller 48 may be present on the wearable 20 to drive LED or fiber optic visual illumination. Examples of such implementations are described in further detail below.

[0038] In operation, the user console 30 may function to convert one or more sensed movements (i.e., of the wearable 20) into one or more audio or visual effects. By chaining together various movements, the user may "play" a sequence of audio or visual effects. FIG. 3 shows a schematic of the method of operation of the present system 10 from the perspective of the user console 30. As shown, the method begins by establishing (at 62) a motion correspondence table that correlates sensed movements with desired audio / visual responses. This step is primarily an initialization step, and in some embodiments may be performed only by loading a pre-established motion correspondence table into memory.

[0039] Once the motion correspondence table is established, the console 30 may receive (at 64) a data stream from the article of footwear or clothing indicative of the wearable's motion. The user console 30 may continuously analyze (at 66) this data stream to identify at least one motion primitive. As used herein, a "motion primitive" is a defined "block" of motion that represents a discrete user action. Examples of possible motion primitives may include sensed linear motion, arcuate motion, sinusoidal / periodic translation, rotation, acceleration, jerk, and / or impact. In a general sense, a motion primitive may include any combination of pre-programmed motion, user-defined motion, and / or automatically detected motion.

[0040] If one or more movement primitives are recognized from the data stream (at 66), the console 30 may trigger the playback of audio samples or visual effects previously associated with the movement primitives (at 68). In some embodiments, the audio samples or visual effects previously associated with the movement primitives may be a sequence of audio samples or visual effects, or a repeating sequence of audio samples or video effects. For example, upon detecting a user stomping their foot (e.g., a movement primitive characterized by a downward velocity followed by a rapid deceleration), the user console 30 may trigger the playback of a single bass beat. Alternatively, it may trigger the playback of multiple bass beats and / or the playback of a looped sequence of bass beats. In this way, the system provides great flexibility for the user to define the audio or video effects that each movement (or sequence of movements) may cause / initiate.

[0041] 4 shows a schematic functional block diagram of the operation of the present system. Each of the described modules or blocks may include computer executable code stored in memory as software or firmware such that when executed, the processor 32 performs the specified functions. In some embodiments, each block may further include any or all necessary hardware that may be needed or may be advantageously used to perform the specified functions. Examples of such hardware may include video or audio encoders / decoders, digital signal processors (DSPs), etc.

[0042] 4, system operation may generally include two distinct modes: initialization 100, and media generation 102. During initialization 100, processor 32 may be tasked with establishing a motion correspondence table 110 that correlates sensed movements with desired audio / video responses. To this end, processor 32 may communicate with at least one of an existing movement database 112, a video source 42, or a motion tracking wearable 20. Prior to building the motion correspondence table 110, processor 32 may first compile a motion catalog 114 of available or expected movements that may be available for later recognition and playback.

[0043] In the simplest embodiment, the movement catalog 114 may simply be imported from an existing movement database 112 that may identify typical or common movements in dance sequences, sports, activities, etc. If a general movement database proves to be too cumbersome (i.e., too many inapplicable movement options) or not specific enough for the desired movement or dance, in some embodiments, the processor 32 may build the movement catalog by analyzing the movements of the connected wearable 20 or by extracting movement information from a provided video source 42.

[0044] In embodiments where motion information is extracted from video, video sources 42 may include, for example, live and locally captured video, pre-recorded video, and / or networked or Internet video feeds / streams. Video sources 42 may be passed through object recognition and tracking module 116 to recognize and estimate the three-dimensional motion of the depicted wearable (or part of an individual's body). In one embodiment, object recognition and tracking module 116 may utilize image processing techniques such as boundary / edge detection, pattern recognition, and / or machine learning techniques to recognize wearable 20 and measure its motion relative to its environment or in a more object-centric coordinate system.

[0045] After processor 32 receives a raw data stream from wearable 20 or recognizes wearable motion depicted from a video stream, it may pass the raw motion through primitive detection module 118. In this module, processor 32 may examine the raw motion for one or more motion primitives or sequences of primitives. For each new primitive or sequence detected, processor 32 may catalog a new general motion or type of motion, a new specific motion, or a new sequence of motions in motion catalog 114. A general motion or type of motion may be, for example, a movement (e.g., any translation) or an impact. A specific motion may be, for example, a specific translation of wearable 20 in a specific direction (e.g., a translation in the medial direction of an article of footwear or an impact of the left foot). Finally, a sequence of motion may be, for example, multiple primitives sequenced together (e.g., a translation in the lateral direction followed by a translation in the medial direction).

[0046] Once the motion catalog 114 has been established, either by direct import, active motion sensing, or video analysis and deconstruction, it may be provided to a user, along with the collection of available audio samples and / or visual effects 120, via an interactive user interface 36. The user interface 36 may receive user input 124 that operates to link one or more cataloged motions (i.e., a type of motion, a specific motion, or a sequence of motions from the motion catalog 114) with one or more audio samples and / or visual effects from the collection of available audio samples and / or visual effects 120. These established relationships between motions and audio samples and / or visual effects may then be stored in a correspondence table 110. In effect, the correspondence table 110 may be a translator that converts future motions into sound or light effects. In addition to creating correspondence between motions and sound / light, the correspondence table 110 may further link one or more motion primitives with haptic responses such that a spectator, if equipped with the appropriate hardware to mechanically communicate with his or her body, feels a predetermined response after a particular motion primitive.

[0047] In some embodiments, the correspondence table 110 need not be a data construct entirely separate from the motion catalog 114, but instead may simply include multiple pointers, each attached to a different motion entry in the motion catalog 114 and referencing a different effect. Additionally, in some embodiments, the correspondence table may further include a set of rules that modify a prescribed output according to timing considerations, such as the rhythm, tempo, or flow of the motion. In this example, timing parameters may be used to change the pitch, tone, tempo, or speed of a given output (or the color, brightness, persistence, or timing of a visual output).

[0048] Once the correspondence table 110 has been created and the initialization 100 is complete, the system 10 may then be set into a media generation mode 102. In the media generation mode 102, the processor 32 may be operative to receive a data stream 130 from the wearable 20 indicative of real-time sensed movement of at least a portion of the wearable 20. The data stream 130 may be received via communication circuitry associated with the processor 32 and made available to the processor in real-time or near real-time. From this received data stream 130, the processor 32 may analyze the movement using a primitive detector 132 (which may be similar or identical to the primitive detection module 118 used during the initialization 100). As described above, the primitive detector 132 may examine the raw movement represented by the data stream 130 and detect one or more movement primitives or sequences of primitives.

[0049] To minimize processing time and therefore improve the responsiveness of system 10, primitive detector 132 may be configured to look only for motion primitives in data stream 130 that have been previously defined in motion catalog 114 and / or assigned with associated audio samples and / or visual effects in correspondence table 110. Upon detecting a primitive, processor 32 may consult correspondence table 110 and then trigger or initiate playback 134 of an audio sample or visual effect in response to at least one identified motion primitive.

[0050] In some embodiments, the collection 120 of available audio samples and / or visual effects may be populated from a pre-existing library that may be provided with the software or downloaded from a connected distributed computing network (e.g., the Internet). However, in one embodiment, a user may populate or add to the collection by directly uploading one or more audio samples from a connected audio source 44 (e.g., a personal computer, a digital music player, a synthesizer, a keyboard, etc.) or by recording one or more sounds or sound sequences that are generated by the system 10 as a result of the user's movements. More specifically, when a user generates a particular beat / rhythm / composition by a movement, the user may save the generated composition in the collection 120 for future playback / triggering by a single individual movement. In this way, it may be possible to layer different sounds / compositions to multiply effects.

[0051] In one configuration, the user can vary the degree of smoothing or auto-tuning between various audio / video effects. In this way, a novice user can create a composition that sounds or looks well-crafted even if the movements are not 100% perfect or in-tempo. Similarly, the wearable 20 can be configured to predict movement primitives based on previous or preceding movements when the user is behind on timing. Conversely, a more advanced user may reduce the smoothing / auto-tuning to have more direct control over the output. In some embodiments, the smoothing / auto-tuning can use machine learning and artificial intelligence techniques to blend the audio / video elements together, which may rely on interspersing additional beats, drawing out notes, predicting subsequent movements based on earlier movement instructions, etc.

[0052] In one configuration, the user console 30 may be configured to take a previously recorded / generated audio track / song and split it into multiple individual segments (automatic segmentation). The nature, duration, and / or segmentation of the various segments may be customized or created separately by the user (manual segmentation / segment modification). The user console 30 may then automatically assign movement primitives to each segment or prompt the user to assign their own movement primitives to each segment. This may result in choreographing a dance routine to a selected song. Then, by performing the scripted movements in the correct tempo, the user (or group of users) may begin playing the song through only those movements (segment by segment). In one embodiment, instead of having an absolute correspondence table, the correspondence table 110 may be conditional on other factors. For example, a given movement primitive may initiate playback of a first audio segment (of a piece of music) if executed between times 00:00:00:00 and 00:00:10, or if executed as an initial move, but the same movement primitive may initiate playback of a second audio segment if executed between times 00:01:40 and 00:02:00, or if it is between the 10th and 20th recognized primitive, or if it follows the recognition of another primitive or the playback of a prescribed audio segment. In further embodiments, various amounts of smoothing / auto-tuning may also be applied to this sequential playback of music segments, while segments may be blended or extended to produce an output in which the last note flows seamlessly from one segment to the next.

[0053] When choreographing to an existing piece of music, the user console 30 may include local memory having the piece of music stored thereon, or the console may be in communication with, for example, an Internet-based streaming audio source to which the user may separately subscribe.

[0054] In addition to outputting a basic sound or beat, or a segment of a pre-recorded track / song, in some embodiments, the correspondence table may include one or more movement primitives linked to audio action / control commands. For example, primitives may be used to start playing a full song, start playing the next song in a list, pause a song, rewind a song, change the beat of a song, change the tone of a song, change the playback volume, fade in / out, etc. In this way, the user may act as a DJ or producer playing pre-recorded audio from a local or networked source through his movements. Additionally, audio action / control commands may operate in conjunction with the display of the user console. For example, primitives may be used to scroll through a list (e.g., to find a song / track).

[0055] For purposes of any of the example applications described herein, it should be understood that any audio playback may include discrete sounds, a collection of sounds, pre-recorded sounds, segments of audio tracks / songs stored in local memory, full audio tracks / songs stored in local memory, segments of audio tracks / songs pulled from an Internet-based source (i.e., including songs that may be accessed via a subscription-based login), or full tracks / songs pulled from an Internet-based source, etc.

[0056] 1, in some embodiments, the system 10 may accommodate multiple users / wearables 20. In such a configuration, each wearable 20 is configured to generate and wirelessly transmit to a user console 30 a respective data stream 130 indicative of the monitored spatial movement of that device.

[0057] As shown generally in Figures 5-7, the system 10 may be configured such that the correspondence table 110 is set globally for all devices (Figure 5), for a group of devices (Figure 6), or on a per-device basis (Figure 7). When set globally, as shown in Figure 5, the console 30 may apply the same set of rules / criteria for each connected wearable (i.e., wearables 20a, 20b, 20c). More specifically, the data stream from each wearable 20a, 20b, 20c may pass through a respective primitive detector 132a, 132b, 132c and may refer to the common correspondence table 110 before playing at 134. Despite there being only a single table, different types of wearables may generate different sounds due to their different intended uses. For example, a watch is unlikely to experience the same impact forces as a basketball shoe.

[0058] FIG. 6 illustrates an embodiment in which the processor 32 maintains different “pods” of wearables (i.e., wearables 20a, 20b, wearables 20c, 20d), each pod having its own correspondence table 110a, 110b (fed by respective primitive detectors 132a, 132b, 132c, 132d). In this embodiment, groups of wearables may be specialized for their purpose. For example, a first grouping of wearables may output primarily bass tones, while a second grouping of wearables may output primarily piano-like tones or visual effects. During initialization, different wearables 20 may “subscribe” to specific pods and thus act as A / V inputs linked to the respective pods.

[0059] Finally, Fig. 7 illustrates a schematic of a multi-wearable use in which each wearable 20a, 20b, 20c is individually provisioned with its own correspondence table 110a, 110b, 110c, respectively. In this multi-wearable environment, the motion primitives for each wearable can be assigned separately from the motion primitives of other wearables, so that similar wearable-specific motions can each result in different outputs. For example, lateral translation of a connected watch can result in the playing of a first sound (e.g., a crash cymbal impact), while a similar translation of a connected article of footwear can result in the playing of a second sound (e.g., a bass drum effect).

[0060] Each of the embodiments shown in Figures 5-7 provides a system where multiple users can collaborate in creating a common piece. In the embodiment shown in Figure 5, each wearable is a separate input to the common system. These devices may generate different movement primitives due to their different nature and use (e.g., an article of footwear may move differently than a watch), but ultimately, all the movement primitives are fed into the same database for output. A dance group may be interested in such a feature to show how they move in sync (i.e., a movement that is off step may have similar results as a movement that is off step). In Figure 7, each wearable may produce a separate suite of sound / visual effects, like a small band made up of different instruments. In this way, multiple users may come together and collaborate on a piece, each with its own tone and tenor. Finally, the embodiment shown in Figure 6 may be similar to a symphony orchestra where there are groups of similar instruments, but each group may have a unique tone and tenor. Ultimately, the technology can be used to create new artistic expressions for movement and can be flexible enough to accommodate both individual and collaborative work.

[0061] 1, in another distributed multi-user scenario, a user's local user console 30 may communicate with one or more remote user consoles 230 via a distributed computing network 50 for the purpose of collaboratively interacting or creating a collaborative audio / visual experience. Thus, the physical proximity of the devices does not limit the limits of a user's creativity.

[0062] Additionally, it should be noted that in each embodiment described herein, some or all of the user console may be physically integrated with the wearable 20. For example, in one configuration, the remote user console 230 may not be a separate computing device, but instead may be a smart / connected wearable 220, such as a smartwatch. In this embodiment, the smart / connected wearable 220 may include sufficient processing and communication capabilities to transmit motion data to a distributed network, and may even communicate with one or more audio output devices 46 or visual effects controllers 48 via a communication protocol such as BLUETOOTH. Similarly, the smart / connected wearable 220 may also function as a user console 30 for one or more other wearables 20 that lack additional processing capabilities.

[0063] Building on this concept of remote users collaborating over a distributed network, in some embodiments the system may be utilized in game-like contexts where users challenge each other to perform and / or create a predefined work of art or to recreate a particular dance sequence. For example, in one context, a user may be presented with a sequence of movements on a display (e.g., an ordered sequence of motion primitives). The user may then attempt to replicate these movements in time, which, if performed correctly, may generate a pre-created audio or visual output. Deviations in the timing or completeness of the user's movements from those intended / displayed may alter the audible or visual output. Additionally, in some embodiments, the processor 32 may be configured to determine from the received data stream an accuracy metric representative of the correspondence between the monitored spatial movements of the wearable 20 and the sequence of ordered motion primitives. The accuracy metric may reflect, for example, deviations in the timing, magnitude, and / or completeness of the user's movements relative to the presented sequence. In some embodiments, the presented sequences may vary in difficulty or complexity based on the user's experience.

[0064] FIG. 8 illustrates an example of such a distributed challenge-based use in a schematic manner. As illustrated, each wearable 20 may directly or indirectly communicate with a distributed computing network 50, such as the Internet. The distributed network 50 may include one or more servers, computers, routers, switches, etc., which may facilitate network interconnectivity, data aggregation, and / or remote application hosting. In one embodiment of a user challenge, a first user 140 (or group of users) may receive a sequenced challenge 142 of movements / actions to perform from a remote computing device 144 (which may be part of the distributed computing network 50) via the user console 30. Following receipt of the challenge 142, the first user(s) 140 may attempt to reproduce the sequence of the ordered movements, or a variation thereof. As with the above-described embodiment, each wearable 20 is configured to generate and wirelessly transmit to the user console 30 a respective data stream 130 indicative of the monitored spatial movements of that device. From this data stream 130, the user console 30 may generate associated audio / visual output and may present the user with a qualitative and / or quantitative scoring 146 of their performance (i.e., accuracy metric 146). The scoring 146 may be calculated by either the user console 30 or the remote computing device 144 and may take into account factors such as deviation in timing from a predetermined beat, completeness of moves from a predetermined magnitude, and / or additional moves that are combined with the base sequence as a means of embellishing the original sequence.

[0065] Following completion of a work or challenge, the user console 30 may transmit an audio or video capture 148 of the work and / or accuracy metrics / scoring 146 from the challenge to a remote computing device 144, which may be hosted for viewing by one or more viewers 150 across the distributed network 50. In addition, the first user 140 may issue a next challenge 152 to a second user 154 directly or via the remote computing device 144 and / or the distributed network 50.

[0066] In another collaborative / competitive example, instead of directly issuing challenges to other users, the technology may be used to bring physically separated users together and / or gamify video-based workouts streamed over the Internet. For example, during an online / streamed kickboxing class, each user watching or listening to the virtual class may be instructed to repeat a sequence of movements or actions. The user's wearable 20 (or connected user console 30) may sense the movement primitives associated with each of the users' movements and may superimpose or cause one or more visual effects to be displayed on the user's display 38 in conjunction with an instructional video. In one embodiment, the color or nature of the visual effect displayed on the display 38 may be altered according to the similarity, completeness, or timing of the sensed primitives compared to those expected or instructed. For example, if the user is achieving a new personal best or exceeding a predetermined accuracy threshold, the user may cause a green star to appear from the top of the display or confetti to rain down. Conversely, the display may provide one or more visual motivators to the user if significant deviations are recognized. In one embodiment, the user's movements may be scored according to their accuracy or completeness when compared to those of the instructor or an idealized standard. This accuracy score may be a running average of accuracy over a predefined period of time and may be displayed on the user's display 38 for personal reference / goal setting. In a situation where multiple users are distributed on a network and each user watches a video, each user's accuracy score may be broadcast (possibly anonymously) to the collective group so that each user knows his or her rank. Similarly, the user's accuracy score may be displayed on the display 38 along with the scores of known influencers, professional athletes, or friends on the social network.

[0067] 9, in some embodiments, spatial movements of one or more wearables 20 may be recorded and uploaded by one or more user consoles 30, 230 to a cloud-based distributed computing network 50 or remote computing device 144. In some embodiments, the visual effects controller 48 may associate one or more visual parameters with the recorded spatial movements so that the movements may be played back or viewed through the augmented reality enabled device 200. In such an example, the nature of the overlaid AR effect (color, trace perception, sequence of visual effects) may be a product of the type of movement primitives that occurred while the spatial movements were being made. The augmented reality enabled device 200 may be a smartphone or AR glasses that operate to overlay a graphical image onto a user's view of the real world (i.e., as perceived through the device) such that the image is perceived by the user to be present at a particular real world location.

[0068] As shown in FIG. 9, in one embodiment, the spatial movement of one or more wearables may be represented by one or more persistent or time-decaying visual traces 202 superimposed on a displayed real-world image. The visual traces 202 may represent, for example, all or part of a half of a basketball court, a game, or a day of movement. In some embodiments, the occurrence of different motion primitives may change the visual effect, color, brightness, or persistence of the visual traces 202. For example, sprinting at high speed down the court may be visualized as an elongated thin visual, while a sharp lateral cut may have a thicker, brighter, or more prominent visual appearance. While FIG. 9 simply illustrates the visual traces 202, in some embodiments there may be corresponding wearable-triggered audio effects, which may be recorded and played back on the video. Thus, a motion primitive that results in a noticeable trace, such as may be caused by dunking a basketball or making a sharp cut, may also have an audio effect associated with it.

[0069] In yet another embodiment, the visual trace may be scored along with music stored in memory associated with the user's device 200 or accessible to the user's device 200 via a streaming media service / subscription. In this embodiment, the user's device may understand the timing and tempo of the movements and the generated visual output and may select an audio / music track with a similar beat or rhythm. If the available audio slightly mismatches the beat or timing of the movements, the user's device may select the closest possible audio and then operate to modify the beat or tempo to match.

[0070] The use of electronic, motion-sensing clothing or footwear in this manner enables new forms of expression and creativity not possible with other, more traditional electronic inputs. Each sport and each athlete within a sporting event can create a unique auditory or visual experience specific to that athlete's style and performance. This experience can be broadcast to users directly (e.g., via speakers or lights within the facility) or via one or more handheld devices in communication with a user console paired with the athlete's wearable. In a non-sporting sense, the technology allows performers (both professionals and those streaming over the internet at home) to use new means of digitally augmented expression, with their own movements being direct triggers of the A / V experience.

[0071] A pre-defined theme or genre may be used or applied in any of the above examples to introduce a new suite of presentations while modifying the sounds or visual presentations according to pre-defined rules. For example, an island / beach theme may change the available sounds to a more calypso-themed suite of sounds and / or change the available colors and / or visual effects to those within a blue / green / brown color palette.

[0072] In one embodiment, as shown in FIG. 10, access to or use of a particular correspondence (or associated theme) may be unlocked and / or made available only if the user device / wearable is present at a particular time and location. For example, members of a flash mob 300 may have access to all predefined A / V themes available upon arrival at the park or square where the flash mob is intended to perform. The user devices / wearables 20 may know their location, for example, through the use of GPS, RF triangulation, wifi network recognition, RF beacons 302, BLUETOOTH location tracking, etc. (collectively, "location sensing means"). Upon arriving at a predefined location (or within a predefined distance thereof) at a predefined time, any wearables 20 registered with the connected server 304 as part of the group may be configured to begin recognizing the user's motion primitives and / or begin transmitting motion data to the distributed network 50. By requiring users to opt-in to a group, participation is limited to only those who knowingly want to be part of the group, while those who happen to be in the right place at the right time are excluded and / or prevented from unknowingly joining the group.

[0073] In the flash mob example, the wearables 20 worn by each user may have connectivity to a common distributed computing network (WAN, WLAN, etc.) to which each wearable is connected either directly or via a smartphone, watch, or other connected device. The translation of sensed movement primitives to triggered responses may be done locally on each wearable 20, or more centrally on a networked server 304 / data aggregation device. The aggregate user output may then be translated into audio or visual output via a local A / V system 306 (e.g., for playback by local speakers 308 or visual effects devices), or passed over the network to a third party device 310 where it may be output as audio (via headphones or speakers) or visually displayed on an augmented or mixed reality screen in a similar spirit to that shown in FIG. 9.

[0074] As mentioned above, in some embodiments, one or more visual effects may be triggered in response to sensed movement primitives. To this end, Figures 11-14 diagrammatically illustrate three embodiments of wearable 20 including one or more integrated light emitting elements 400 that may be triggered to project visible light outwardly in response to one or more detected movements / movement primitives (e.g., the direction of visual effects controller 48).

[0075] As shown generally in FIG. 11, in one embodiment, the light emitting element 400 may be a projector 402 that operates to project a predetermined pattern of light onto an adjacent surface 404, such as the ground. As specifically shown, the projector may include a housing 406, a light source 408, such as a light emitting diode (LED), one or more lenses 410 (e.g., one or more focusing and / or objective lenses), and an optional slide or stencil 412. If used, the slide or stencil 412 allows the projector 402 to project an image (e.g., a logo, silhouette, or graphic) onto the surface 404, rather than simply projecting a regular ball of light. In one configuration, the projector 402 may be integrated into a midsole 420 of an article of footwear 422 and oriented to project light through an outsole or ground-facing outsole surface 424. In other embodiments, instead of projecting light through the outsole or the ground-facing outsole surface 424, the projector may be positioned / oriented to project an image onto the ground surface 404 next to the article of footwear 422. In such embodiments, the projector 402 may be positioned on or extend at least partially through a sidewall of the midsole.

[0076] In a configuration in which the light emitting elements 400 shine through the ground-facing outsole surface 424 of the article of footwear, the projected light is visible to a surrounding observer only when the wearer of the article lifts his or her foot off the ground. To conserve power, in one configuration, the light emitting elements 400 may be controlled to illuminate only if the article of footwear detects that it is moving away from the ground, or if the foot is detected to be moving a sufficient distance away from the ground such that the emitted light is visible to an observer (i.e., where position may be derived from sensed acceleration data). In yet another embodiment, the light emitting elements 400 may be controlled to illuminate only if the wearable 20 / motion sensing circuitry 22 senses an upward acceleration above a certain threshold. Such may indicate a jump or a jump of a particular magnitude, and may further help ensure a certain minimum duration for illumination, or ensure that an ideal focal length is achieved for the projector 402 to project a sufficiently clear image.

[0077] As shown in FIGS. 12-13, in some embodiments, the light emitting element 400 may be a lighted panel 440 and / or an illuminated design element 442 provided on or otherwise integrated into an article of footwear or clothing (i.e., an illuminating article 444). In some embodiments, this illumination article 444 may be the same as the wearable 20 described above, while in other embodiments, it may be distinct from the wearable 20 by being in electrical communication (directly or indirectly) with the wearable 20. For example, the wearable 20 may be an article of footwear and the illumination article 444 may be a glove or a shirt. The article 444 may further include a power source 446 and a controller 448 operative to modulate the power provided from the power source 446 to the panel 440 and / or the design element 442. In some embodiments, the controller 448 may be the visual effects controller 48 described above.

[0078] 12 generally illustrates illumination article 444 as an article of footwear 460 including an upper 462 coupled to a sole structure 464. Sole structure 464 extends between the foot and the ground when article 460 is worn. In different embodiments, sole structure 464 may include various cushioning components including a foamed polymer midsole, one or more integrated fluid-filled chambers (e.g., airbags), a foam insole, or the like. Additionally, in some embodiments, sole structure 464 may include one or more recesses or cavities to house power source 446, controller 448, and / or any other sensors or electronic components that may be used by article 444.

[0079] Upper 462 may include various provisions for containing and covering the foot as well as for fastening article 444 to the foot. Upper 462 includes an opening 466 for the foot to enter the interior cavity of upper 462. Some embodiments may include fastening provisions including, but not limited to, laces, cables, straps, buttons, zippers, as well as any other provisions known in the art for fastening an article.

[0080] As generally shown in FIG. 12, in one embodiment, the exterior wall of the upper 462 may include a lighted panel 440. The lighted panel 440 may include multiple different layers, at least one of which is an electroluminescent layer (or EL panel). Exemplary EL panel technologies that may be used include, but are not limited to: light emitting capacitor (LEC) panels, powder phosphor-based electroluminescent panels, and thin-film electroluminescent materials. Additionally or alternatively, the lighted panel 440 may include one or more light emitting diodes or other lighting elements that, when diffused, may illuminate an area of ​​the panel. Further embodiments and further details regarding an article 444 having a lighted panel 440 are described, for example, in U.S. Pat. No. 10,182,608, which is incorporated by reference in its entirety.

[0081] In some embodiments, the article of footwear 460 may include one or more illuminated design elements 442. These design elements 442 may include spot-lit features or individual elements rather than a more broadly illuminated area, such as light panels 440. However, in some embodiments, there may be overlap between the two (i.e., the design element may be a panel illuminating logo or accent). In this case, the one or more illuminated design elements 442 may include an illuminated logo or decoration as described in U.S. Pat. No. 8,056,269, an illuminated strap as described in U.S. Pat. No. 10,004,291, an illuminated strand as described in U.S. Pat. No. 8,813,395, and an illuminated cavity or fluid-filled cushioning component as described in U.S. Pat. No. 8,356,430 and U.S. Patent Application Publication No. 2009 / 0158622, each of which is incorporated by reference in their entirety.

[0082] While FIG. 12 illustrates the article 444 as an article of footwear, FIG. 13 illustrates the article 444 as an article of apparel / clothing 480, more specifically, as a shirt or jacket as it is worn. In other embodiments, the apparel / clothing 480 may additionally or alternatively include other types of shirts (long sleeve, short sleeve, tank top, vest), pants, socks, body suits, hats, gloves, outerwear, and / or any other desired type of apparel. As shown, the apparel / clothing 480 includes a light array 482 including the illuminated design element 442, a plurality of embedded conductive traces 484, a power source 446, and a controller 448. The light array 482 may include, for example, a plurality of separate visible light sources (e.g., LEDs, OLEDs, electroluminescent materials, etc.) arranged in a single line of light or in two or more columns of light. The light array 482 may also include, for example, a continuous line of light using fiber optic light and / or optical fibers. The light array 482 is configured to be thin, bendable, and flexible to conform to the curvature of the body when the apparel item is worn. The light array 482 can be secured to the apparel product using a polymer layer, such as, for example, a thermoplastic polyurethane (TPU) or a silicone-based polymer. Further details regarding exemplary light-emitting articles of apparel / garment are described in U.S. Patent Application Publication Nos. 2019 / 0059461 and 2019 / 0200690, both of which are incorporated by reference in their entireties.

[0083] In some configurations, the light emitting elements 400 of FIGS. 12-13 may be used as one or more of the outputs of the user collaboration architecture shown in FIGS. 5-7. For example, in a collaborative visual or audio / visual production, multiple users may each wear one or more lighting articles 444 having one or more light emitting elements 400 disposed thereon. Different motion primitives or combinations of motion primitives may then trigger different individual illuminations of the light emitting elements 400, with the color, luminosity, duration, and / or lighting pattern (e.g., solid, blinking, fast blinking, scrolling, etc.) predefined in response to the motion primitive or otherwise a function of the magnitude of the primitive. For example, a sharper lateral cut may result in a brighter panel illumination than a less aggressive lateral cut (e.g., as measured by the magnitude of lateral acceleration or the magnitude of lateral jerk (da / dt)).

[0084] In one example, a body suit or jacket as shown in FIG. 13 may be used by a stage performer who wants to add visual flair to their performance. In another example, lighting articles of apparel as shown in FIGS. 12 and 13 may be used by a runner or cyclist to signal others nearby. For example, a runner with a smartwatch wearable may raise their hand to signal oncoming traffic. The wearable 20 may recognize a movement primitive associated with lifting the hand, which may then trigger some or all of the light emitting elements 400 to light up. In one configuration, different hand movements may cause different lighting patterns. For example, extending the hand upwards may light up a first set of lights, while extending the hand outward may light up a second set of lights.

[0085] When using the present technology to create an audio / visual performance, staying on beat in the absence of external rhythms can be particularly difficult. More specifically, in such use cases, the user may lack many external cues (e.g., base beats) that could otherwise be used to set the rhythm / timing. This problem can be particularly evident in collaborative group productions that may otherwise rely on the beat of a song to synchronize the performers' tempo and rhythm.

[0086] To aid one or more users in beat tracking and / or maintaining synchronization with other collaborating individuals, the wearable 20 may include a vibration transducer 500 that operates to convey tactile sensations to the user's body. FIG. 14 illustrates one embodiment of such a wearable device 20. In this embodiment, the vibration transducer 500 is provided within a sole structure 464 of an article of footwear 444. However, in other embodiments, the vibration transducer 500 may be provided in an upper 462 of the article of footwear 444, or on / in an article of clothing, such as a watch, sleeve, shirt, pants, shorts, gloves, hat, or other garment or accessory in contact with the user's body.

[0087] The vibration transducer 500 may operate at the direction of the processor 28 to communicate a beat or other tactile timing signal to the user. In a multi-user environment, this timing signal may then be synchronized across each of the multiple users to help create a properly timed production (e.g., during a flash mob). In one configuration, the vibration communicated to the user by the vibration transducer 500 may be a switched or compound vibration that includes a tactile waveform that switches according to a particular beat. The tactile waveform may be, for example, but not limited to, a vibration having a frequency in the range of about 100 Hz to about 300 Hz, or in the range of about 140 Hz to about 210 Hz, or in the range of about 230 Hz to about 260 Hz. The vibration may be selected so that the user can most easily perceive the notification from the vibration transducer 500 when such notification is provided.

[0088] The intended beats of a song or piece may be represented by periodic transmissions of haptic waveforms with periodic transmissions having a duty cycle of less than about 50%, or more preferably less than about 30%, or in the range of about 5% to about 25%. The beats may have a transmission frequency of about 0.1 Hz to about 3 Hz. More appropriately, when measured in beats per minute (BPM), the switched beats may transmit about 30 to about 160 individual haptic waveforms per minute. In some embodiments, all beats of a piece need not be represented. Rather, only certain synchronous beats (e.g., one of four or one of eight consecutive beats in a piece) may be represented. Regardless of the particular frequency of the beats, the haptic waveforms may represent short-duration buzzes, and the beats are the timings at which such short-duration buzzes occur.

[0089] In some embodiments, instead of the periodic vibration transmitted by a vibration transducer, a similar beat may be transmitted by contracting or tightening a portion of the wearable 20 (or an article of footwear or clothing in communication with the wearable 20) around the user's body. To achieve contraction, the wearable 20 (or other article) may include one or more tensioning mechanisms that operate to pull one or more fibers, cords, laces, closure mechanisms, or other fit-adjusting aspects of the article in a tangential direction around a portion of the wearer's body. In doing so, tension in the article may urge the article to contract or reduce in size in a radial dimension, which exerts a compressive force on the wearer. Articles that may be particularly suitable for such compression include shoes (adaptive lace tensioning), compression sleeves / garments, and watches / bracelets.

[0090] In some embodiments, the tensioning mechanism may include a motor that operates to wind / unwind a tensioning fiber embedded within the article in response to an actuation signal. In other embodiments, the tensioning mechanism may include one or more linear actuators, fast response active materials (e.g., piezo actuators), or microelectromechanical systems (MEMS). The tensioning mechanism may be configured to respond to an applied switched beat to periodically induce momentary compression against the user's foot / body. Further description of tensioning mechanisms in the context of footwear is provided in U.S. Pat. No. 10,448,707, U.S. Patent Application Publication Nos. 2018 / 0199673, 2017 / 0265583, and / or U.S. Patent Application No. 16 / 694,306, each of which is incorporated by reference in its entirety.

[0091] FIG. 15 illustrates a schematic of a charging device 600 that may be used with the wearable 20 as described above. As will be appreciated, each wearable 20 may include a battery that requires periodic recharging to maintain the device's ability to perform the functions described above. To provide a waterproofing measure, the charging device 600 may be an inductive charging device comprising one or more inductive coils 602. Each inductive coil 602 may be operative to magnetically induce a charging current within the wearable 20 when the coil 602 is energized with an AC waveform and brought into close proximity to the wearable 20. In the case of an article of footwear, two inductive charging packs 604, 606 may be required (i.e., one for each shoe), with each pack 604, 606 including an energizable inductive coil 602. An exemplary charging device is described in U.S. Patent Application Publication No. 2017 / 0150773, which is incorporated by reference in its entirety.

[0092] As shown diagrammatically in FIG. 16, in some embodiments, the charging device 600 may further include disco ball or party light capabilities. More specifically, each charging pack 604, 606 may include a number of lighting elements 608 (e.g., LEDs) operative to project light of varying intensity and / or color. Additionally, the charging device 600 may include mounting features 610, such as a hook or stand, that may support the charging device in a party-friendly condition. The mounting features 610 may be integrated with or otherwise attached to a central hub 612 from which each of the charging packs 604, 606 extends. The central hub 612 may include an energy storage device 614, a movement mechanism 616, and / or a lighting controller 618, while also providing suitable strain relief for any wires extending to the charging packs.

[0093] When used as a party light, the energy storage device 614 (e.g., a battery) may provide the power needed to illuminate the lighting elements 608 as well as the power needed to facilitate external communication or drive the motorized mechanisms of the locomotion mechanism 616. The energy storage device 614 may be charged simultaneously with the wearable 20 when the charging device is plugged into an AC power source. Conversely, the energy storage device 614 may consume energy when the charging device 600 operates as a party light.

[0094] The movement mechanism 616 may be configured to induce movement of the charging packs 604, 606 and / or the lighting elements 608 to provide a visual effect similar to a disco ball or other moving spotlight. The movement mechanism may include, for example, one or more motors, winding / spring drive mechanisms, or articulating mirrors / lenses. The movement mechanism 616 may generally generate a rotation or vibration of at least a portion of the charging pack or lighting element to change how the light is projected. As shown generally in FIG. 16, in one particular embodiment, the movement mechanism 616 may rotate the central hub 612 relative to the mounting feature 610, which may rotate the charging packs 604, 606 about a central axis 620.

[0095] The lighting controller 618 may be responsible for lighting one or more lighting elements 608 to create a visual effect. In one configuration, the lighting controller 618 may simply light the lighting elements 608 in a repeating pattern, which may be preprogrammed and / or user controlled. For example, a predetermined collection of lighting patterns and sequences may be preprogrammed into the lighting controller at the time of manufacture. The user may then select a desired pattern / sequence, for example, by clicking a button on the hub 612 or by selecting it via a connected wireless device (e.g., a remote, smartphone, computer, etc.).

[0096] In another configuration, the lighting controller 618 may illuminate the lighting elements 608 in response to audio that may be sensed via an integrated or connected microphone. The lighting controller may, for example, pulse lights in response to a sensed low frequency beat, or change lighting patterns or colors according to a sensed pitch or frequency. In one configuration, the microphone may be integrated into a connected wireless device that may transmit either the sensed raw audio data, audio summary data (e.g., beat or pitch data), or program commands based on the sensed raw audio or audio summary data.

[0097] In yet another configuration, the lighting controller 618 may receive a digital signal from the wearable 20 and may illuminate the lighting elements 608 based on one or more sensed movement primitives, as described above. As such, the lighting controller 618 may include or be in communication with the visual effects controller 48, and the charging device 600 may serve as a means for displaying a visual representation of the user's movements (or may otherwise complement the user's movements). In this manner, it may be possible for the lighting elements 608 to synchronize their movements and / or flash with the rhythmic movements of one or more people dancing near the light.

[0098] In some embodiments, instead of triggering the playback of an audio sample or visual effect in response to at least one identified movement primitive, the movement primitive may serve as an input to an external game. For example, in some embodiments, the shoes may serve as an input to a dance-based video game in which a sequence of movements is displayed on a video display, and the user must do their best to replicate the movements to achieve a high score. In another embodiment, as shown in FIG. 17, the game may be a retail-centric game that requires the user to perform one or more sequences of movements to acquire an unlock code, token, pass, authorization, or opportunity to acquire a retail product or digital collectible. In some implementations, the retail product or digital collectible may be a limited release retail product, a digital representation of a retail product, a cryptographically protected digital asset, or other item with limited or restricted supply, as described in U.S. Patent Application Nos. 15 / 931,764 and / or 16 / 707,720, both of which are incorporated by reference in their entireties.

[0099] FIG. 17 generally illustrates an interactive retail kiosk 700 that requires active user engagement in order for a user 702 to obtain a retail product 704, a digital representation of the retail product, an unlock code or portion of an unlock code to obtain a physical or digital retail product, or digital recognition of an achievement (e.g., a virtual badge that can be shared on social media).

[0100] FIG. 18 illustrates generally a method 800 of acquiring a physical or electronic object or code, such as may be performed in the interactive retail kiosk 700 of FIG. 17. As illustrated, the method may begin (at 802) by recognizing the presence of a user 702 within a predefined physical proximity of the kiosk 700. In one configuration, the kiosk 700 may include a camera, an RFID reader, a pressure mat on the floor next to the kiosk 700, or a button that the user presses to alert the system to the user's proximity. In another configuration, the user 702 may possess a wireless computing device 706 in wireless digital communication with a server node 708 over a communications network 710. The wireless computing device 706 may have a means for recognizing the user's location, such as a GPS receiver, a wireless communication component capable of recognizing the device's proximity to a wireless beacon 302 or local area network, or a camera capable of perceiving and recognizing visual aspects or digital codes in the surrounding environment. Once the user's location is known, an application running on the device may send an indication of the user's relative or absolute location in the vicinity of the kiosk 700 to the server node 708, which may initiate a game.

[0101] The wireless computing device 706 may further include the capability to authenticate the user's identity, such as by including an application that requires the user to enter a password, present a discernible biometric feature (e.g., fingerprint or facial recognition), or otherwise securely log into the application. At step 804, the user's wireless computing device 706 may authenticate the user's identity and location as physically present within a predefined proximity or geofence of the kiosk 700.

[0102] Once the user is authenticated and recognized in the vicinity of the kiosk 700, the server 708 may present (at 806) the user with a particular challenge. In one configuration, the challenge may be presented to the user 702 via a display 712 coupled to or in communication with the kiosk 700. In another embodiment, the challenge may be presented to the user via the communications network 710 and a display provided on their wireless computing device 706. In general, the challenge may include a series of movements, actions, or actions that the user is requested to perform in sequence. For example, the challenge may include a dance sequence, a set of exercises such as jumping jacks, lunges, burpees, high knees, kicks, or a series of yoga poses.

[0103] Once the user is presented with the challenge at 806, the user 702 may perform the requested physical activity and the server node 708 may determine (at 808) that the challenge has been successfully completed. This determination may come from either direct observation (e.g., via a camera or pressure mat provided on the kiosk 700) or from receiving a digital indication that the challenge has been performed. In one configuration, the received digital indication may be provided by the wearable 20 as a transmitted sequence of movement primitives, or as a simple indication that the wearable 20 has detected a sequence of movement primitives that matches or approximates those expected during successful completion of the challenge.

[0104] Following a determination that the user 702 has successfully completed the challenge (at 808), the kiosk 700 presents the user with a code (at 810) that may be redeemed for a retail product (at 812) or combined with other codes to be collectively redeemed for a retail product (or limited release digital object). As generally shown in FIG. 17, in one embodiment, the kiosk 700 may provide the user 702 with a printed receipt 714 having a machine-readable code 716 printed thereon. In another embodiment, the kiosk 700 may digitally transmit the code 716 to the user device 706 or to a user account that was authenticated by the server 708 prior to the issuance of the challenge. Once transmitted, the code 716 may be maintained in a digital locker / storage or cold storage digital wallet to allow the user to access the code at a future time.

[0105] In one embodiment, the kiosk 700 may be located in a retail store and the code may be handed to a retail associate or entered into an internet interface to redeem a retail product of a similar style or design to the product 704 displayed on the kiosk 700. In another embodiment, the user may need to obtain multiple codes, each from a different kiosk, to qualify for the product. In this multi-kiosk scenario, multiple kiosks may be distributed across a geographic region and / or multiple retail stores, and the user may be required to search for them, as described in U.S. Patent Application Publication No. 2019 / 0213619 and U.S. Patent Application No. 16 / 874,944, which are incorporated by reference in their entireties. In this configuration, the user may be sent on a scavenger hunt to accumulate the codes necessary to unlock the product. As generally described in U.S. patent application Ser. No. 16 / 874,944, in one embodiment, the user may be guided to the next kiosk through the use of turn-by-turn navigation built into the shoe (e.g., by selectively pulling on the laces or by selective actuation of the tactile transducers shown in FIG. 14).

[0106] 17 as a physical product display, in other embodiments it may simply be a non-retail display, such as an oversized logo, sign, or other marker that may or may not have direct interactivity. In some embodiments, kiosk 700 may be a virtual kiosk (i.e., viewable with a suitable wireless computing device in an augmented or mixed reality context) or may be a location such as a virtual waypoint or geofenced area.

[0107] While most of the above disclosures have focused primarily on generating audio and / or visual effects within a live environment, similar experiences can also be generated within virtual worlds (e.g., as shown in FIG. 19) or used to enhance video broadcasts of live events (e.g., as shown in FIG. 20).

[0108] Referring to FIG. 19, it is becoming increasingly common for musical artists 900 to broadcast live concert performances within online virtual environments. Such virtual environments 902 may be hosted and / or maintained by a game server 904 connected to a distributed computing network 50, such as the Internet. Multiple users may interact with the virtual environment and with each other through unique user-controlled avatars 906 within the environment 902. During gameplay, each avatar 906 may respond to respective movement / action commands made by a web-enabled computing device 908 controlled by a different respective user 910. Each user 910 may experience a first-person or third-person perspective from / of the avatar 906 within the virtual environment 902 via a display associated with the respective web-enabled computing device 908. A popular example of such an environment exists through the FORTNITE® game developed by Epic Games.

[0109] In the embodiment shown in FIG. 19, very similar to the embodiments described above, the user console 30 may receive a data stream from a wearable 20 mounted on the body of a musical performer 900. The user console 30 may rely on a similar motion match table 110 to trigger the playback of an audio sample or visual effect that was previously associated with that motion primitive. However, the main difference is that the triggered audio sample or visual effect occurs only within the virtual environment 902. For example, a lateral swish of the performer's foot may cause the sky in the environment to change color, a stomp of the user's foot may cause virtual fireworks to go off, and a downward thrust of the user's fist may cause the playback of a strong percussion / kick drum, which may be accompanied by a visual shock wave emanating outward from the performer's avatar 912, for example. Additionally, instead of movement primitives being used only to control sound or visual effects, the movement primitives sensed by the performer's wearable(s) may also be used to control or modify some or all of the movements or actions of the performer's avatar 912 within the virtual environment 902.

[0110] While FIG. 19 utilizes a wearable 20 to control aspects of a virtual world, the system 950 shown in FIG. 20 operates to use sensed motion to enhance one or more aspects of a live video broadcast. In the context of a sporting event, a portion of an athlete's uniform may include a wearable 20 that operates to detect or extract one or more motion primitives from the wearer's movements / activities. When an athlete executes a predetermined move or sequence of moves, a visual broadcast 952 (television, cable, virtual reality, internet stream, etc.) of the sporting event may then be augmented / modified with one or more visual effects 954 to highlight or accentuate the athlete's actions. For example, in a basketball game, when an athlete attempts to dunk the ball with a threshold amount of force or leaping power, a broadcast system 956 may overlay a video feed 958 with a graphic of an explosion emanating from the athlete himself on the court surface as he hits the ground. In another example, when a basketball player executes a signature crossover move, the sequence of actions may cause the broadcast system 956 to graphically change the perceived contours of the floor in the video feed 958, as if the athlete's plant legs were creasing the court surface. In so doing, this use of the technology injects aspects of a decorative or cartoony video game (e.g., similar to NBA JAM by Electronic Arts Inc.) into live-action sports, triggered by the directly sensed movements of the athlete himself (rather than by a third party visual observer).

[0111] In some embodiments, in addition to motion triggers, the generation of visual effects may be further conditioned on external factors such as in-game timing, recent scoring activity, or the relative positioning of other players on the court / field. Furthermore, in some embodiments, the presence, nature, or color of the effects may vary based on one or more preferences or attributes provided by the viewing user (i.e., via the user's respective web-enabled computing device 960). In such embodiments, the user may first pre-select a favorite team or player, which may dynamically assign different motion correspondence tables to different players / teams. During the video broadcast, the pre-selected favorite team or player may then be augmented by a first set of sounds / graphics, and the other players (and, in particular, the opposing player / team) may be augmented by a second set of sounds / graphics. The difference in treatment between the designated favorite player / team and the opponent may have the effect of casting the pre-selected favorite player as the hero and the opposing player / team as the villain. This may be accomplished, for example, by the use of different color palettes, different graphics, and / or different sound effects.

[0112] Similar to enhanced sports broadcast 952, the visual effects implanted in a virtual environment 902 as shown in FIG. 19 may vary depending on one or more attributes or preferences of users within that world. For example, the clothing / skin worn by a user's avatar may unlock certain color schemes or visual effects. Similarly, the level or experience of a user's avatar may also unlock different visual or audio effects. This may occur, for example, by applying entirely different match tables or by making aspects of the match tables conditional on certain criteria.

[0113] In addition to simply varying the visual or audio effects, it is also possible for the output of the correspondence table to be transmitted to the user in the form of one or more haptic signals. Each haptic signal may instruct a haptic device on the user's body to provide a haptic response. These haptic responses may be in the form of vibrations or contractions of an article of footwear or clothing placed on the user and may be synchronized with one or more of the visual or audio effects. In this manner, the technology may be delivered by multiple senses to provide a more immersive user experience. In one embodiment, the haptic signal may further attempt to convey or impart one or more haptic sensations that may be similar to those experienced by a live athlete.

[0114] Various features and methods of operation of the presently described technology are set forth in the following clauses:

[0115] Clause 1: A system for motion-based media creation comprises: an article of footwear or clothing having at least one accelerometer or inertial measurement unit operative to monitor spatial movement of at least a portion of the footwear or article of clothing and generate a data stream indicative of the monitored spatial movement; and a processor in wireless communication networked with the footwear or article of clothing, the processor configured to: receive the data stream from the article of footwear or clothing; identify at least one movement primitive from the received data stream; and trigger playback of an audio sample or a visual effect in response to the identified at least one movement primitive.

[0116] Clause 2: The system described in clause 1, wherein the identified at least one movement primitive includes a first movement primitive and a second movement primitive, and the audio sample or visual effect is a first audio sample or a first visual effect and is triggered in response to the first movement primitive; the processor is further configured to: trigger playback of a second audio sample or a second visual effect in response to the identified second movement primitive; the first audio sample or first visual effect is different from the second audio sample or second visual effect.

[0117] Clause 3: The system of any of clauses 1-2, further comprising a user input device and a user input in communication with a display, wherein the processor is further configured to: maintain a library of a plurality of audio samples; associate a selected audio sample from the plurality of audio samples with a predefined motion primitive based on input received from the user input device; match the identified motion primitive with the predefined motion primitive; and triggering the playback of the audio sample or the visual effect in response to the identified motion primitive includes outputting the selected audio sample in response to matching the identified motion primitive with the predefined motion primitive.

[0118] Clause 4: The system of any of clauses 1-3, further comprising a user input device and a user input in communication with a display, and further configured to: maintain a library of a plurality of visual effects; associate a selected visual effect from the plurality of visual effects with a predefined motion primitive based on input received from the user input device; match the identified motion primitive to the predefined motion primitive; and wherein triggering the playback of the audio sample or the visual effect in response to the identified motion primitive includes outputting the selected visual effect in response to matching the identified motion primitive with the predefined motion primitive.

[0119] Clause 5: The system of clause 4, wherein the footwear or article of clothing further has at least one light; and outputting the selected visual effect includes illuminating the at least one light of the footwear or article of clothing.

[0120] Clause 6: A system as described in any of clauses 1 to 5, wherein the article of footwear or clothing is an article of footwear.

[0121] Clause 7: The system of any of clauses 1 to 6, wherein the user console is in communication with a distributed network; the processor is further configured to: receive a challenge from the distributed network, the challenge comprising an ordered number of motion primitives; display the challenge to a user via a user display device; and determine from the received data stream an accuracy metric representative of correspondence between the monitored spatial motion of the footwear or article of clothing and the ordered number of motion primitives.

[0122] Clause 8: The system of clause 7, wherein the processor is further configured to transmit the accuracy metric to the distributed network.

[0123] Clause 9: A system described in any of clauses 1 to 8, wherein the processor is configured to trigger the playback of the audio sample or the visual effect in a virtual environment.

[0124] Clause 10: The system of clause 9, wherein the playback of the audio samples or the visual effects is movement of an avatar within the virtual environment.

[0125] Clause 11: A system described in any of clauses 1 to 8, wherein the processor is configured to trigger the playback of the audio sample or the visual effect as an overlay on a video feed.

[0126] Clause 12: A system described in any of clauses 1 to 11, wherein the processor is further configured to determine a distance between the footwear or article of clothing and a predetermined location, and the processor is configured to trigger the playing of an audio sample or visual effect only if the distance is below a predetermined threshold.

[0127] Clause 13: A method of creating an entertainment experience from a user's movements, the method comprising: receiving a plurality of data streams, each data stream being from a different article of footwear or clothing and representing spatial movements of a respective said article; and triggering the playing of an audio sample or a visual effect in response to each of the received plurality of data streams.

[0128] Clause 14: The method of clause 13, further comprising identifying a plurality of movement primitives from the plurality of data streams, and wherein triggering the playback of the audio samples or visual effects comprises triggering the playback of different audio samples or visual effects for respective different ones of the identified plurality of movement primitives.

[0129] Clause 15: The method of clause 14, further comprising mapping a plurality of pre-defined movement primitives to a plurality of different audio samples or visual effects.

[0130] Clause 16: The method of clause 15, wherein the mapping step includes mapping each of the plurality of movement primitives to a different one of the plurality of different audio samples or visual effects.

[0131] Clause 17: The method of any of clauses 14 to 16, further comprising: receiving a challenge from a distributed network, the challenge comprising an ordered set of motion primitives; displaying the challenge to one or more users via one or more user display devices; and determining an accuracy metric representative of correspondence between the identified set of motion primitives and the ordered set of motion primitives from the received data streams.

[0132] Clause 18: The method of any of clauses 13-17, further comprising: maintaining a library of a plurality of audio samples and a library of a plurality of predefined motion primitives; associating a different selected audio sample from the plurality of audio samples with each of the plurality of predefined motion primitives based on input received from a user input device; identifying a plurality of motion primitives from the plurality of data streams; and matching each of the identified motion primitives with one of the predefined motion primitives; wherein the step of triggering the playback of the audio samples or visual effects comprises the step of outputting the audio samples associated with each of the predefined motion primitives that match an identified motion primitive.

[0133] Clause 19: The method of any of clauses 13-17, further comprising: maintaining a library of a plurality of visual effects and a library of a plurality of predefined motion primitives; associating a different selected visual effect from the plurality of visual effects with each of the plurality of predefined motion primitives based on input received from a user input device; identifying a plurality of motion primitives from the plurality of data streams; and matching each of the identified motion primitives with one of the predefined motion primitives; wherein triggering the playback of the audio samples or visual effects comprises outputting the visual effect associated with each predefined motion primitive that matches an identified motion primitive.

[0134] Clause 20: The method of clause 19, wherein the step of outputting a visual effect includes the step of illuminating at least one light on the article of footwear or clothing.

[0135] Clause 21: The method of any of clauses 13 to 20, wherein receiving the plurality of data streams comprises receiving the data streams via a wireless digital communications protocol.

[0136] Clause 22: The method of any of clauses 13 to 21, further comprising: identifying a first motion primitive from a first data stream of the plurality of data streams and identifying a second motion primitive from a second data stream of the plurality of data streams, wherein the first motion primitive and the second motion primitive are the same; and triggering the playback of the audio sample or the visual effect comprises triggering the playback of a first audio sample or visual effect in response to identifying the first motion primitive and triggering the playback of a second audio sample or visual effect in response to identifying the second motion primitive; wherein the first audio sample or visual effect is different from the second audio sample or visual effect.

Claims

1. 1. A method of operating an interactive retail kiosk for distributing product redemption codes for distribution of physical or digital products, the method comprising: the interactive retail kiosk identifying the presence of a user within a predefined physical proximity of a location; the interactive retail kiosk authenticating the identity of the user; the interactive retail kiosk presenting a physical challenge to the user; the interactive retail kiosk electronically determining that the physical challenge has been performed by the user; and the interactive retail kiosk presenting a product redemption code to the user following a determination that the physical challenge has been performed by the user; the physical challenge comprises a series of movements, actions, or movements that the user is required to perform in sequence; The step of the interactive retail kiosk electronically determining that the physical challenge has been performed by the user includes: the interactive retail kiosk receiving a digital indication from a wearable computing device mounted on the user's body, the wearable computing device operable to monitor the user's body movements, the digital indication including an indication that the wearable computing device detected a sequence of movement primitives that matches or closely approximates expected movements for successfully completing the challenge; and determining, by the interactive retail kiosk, from the digital representation provided by the wearable computing device, whether the physical movements of the user include the sequence of movements, movements, or actions that the user is asked to perform in sequence. How it works.

2. The step of the interactive retail kiosk identifying the presence of the user within the predefined physical proximity of the location includes: said interactive retail kiosk electronically detecting said presence of said user with a digital camera, an RFID reader, and a pressure mat provided at said location; The method of claim 1 .

3. The step of the interactive retail kiosk identifying the presence of the user within the predefined physical proximity of the location includes: the interactive retail kiosk wirelessly receiving from the user's wireless computing device a digital representation of the user's relative or absolute location; The method of claim 1 .

4. and wherein the interactive retail kiosk authenticating the identity of the user includes the interactive retail kiosk receiving a password or a biometric characteristic from the user via a digital application running on a wireless computing device. The method of claim 1 .

5. the wearable computing device comprises an article of footwear or apparel; The method of claim 1 .

6. the interactive retail kiosk receiving the product redemption code from the user; and the interactive retail kiosk providing the user with a physical or digital object in exchange for the received product redemption code. The method of claim 1 .

7. the digital object is a cryptographically secured digital collectible; 7. The method of claim 6.

8. The step of the interactive retail kiosk presenting a product redemption code to the user includes: the interactive retail kiosk printing a machine readable code on a printed receipt; The method of claim 1 .

9. The step of the interactive retail kiosk presenting a product redemption code to the user includes: the interactive retail kiosk digitally transmitting the product redemption code to a portable computing device; The method of claim 1 .

10. the interactive retail kiosk receiving a plurality of product redemption codes from the user, each product redemption code corresponding to a different one of a plurality of physical challenges; the interactive retail kiosk providing the user with a physical or digital object in exchange for the plurality of product redemption codes received; Further comprising: The method of claim 1 .

11. The location includes a physical product display, a non-retail display, a virtual waypoint, or a geofenced area. The method of claim 1 .

12. the interactive retail kiosk further comprising displaying a virtual kiosk to the user in augmented reality or mixed reality. The method of claim 1 .

13. an interactive retail kiosk identifying a presence of a user within a predefined physical proximity of the retail kiosk; the interactive retail kiosk authenticating the identity of a user via at least one of the retail kiosk or a portable computing device in digital communication with the retail kiosk; the interactive retail kiosk presenting a physical challenge to the user via a display in digital communication with the retail kiosk or via the portable computing device, the physical challenge comprising a series of movements, actions or actions that the user is asked to perform in sequence; the interactive retail kiosk electronically determining that the physical challenge is performed by the user; and the interactive retail kiosk presents to the user a retail product, a digital representation of a retail product, a digital collectible, a product redemption code, or a portion of a product redemption code following a determination that the physical challenge has been performed by the user; the physical challenge comprises a series of movements, actions, or movements that the user is required to perform in sequence; The step of the interactive retail kiosk electronically determining that the physical challenge has been performed by the user includes: the interactive retail kiosk receiving a digital indication from a wearable computing device mounted on the user's body, the wearable computing device operable to monitor the user's body movements, the digital indication including an indication that the wearable computing device detected a sequence of movement primitives that matches or closely approximates expected movements for successfully completing the challenge; and determining, by the interactive retail kiosk, from the digital representation provided by the wearable computing device, whether the physical movements of the user include the sequence of movements, movements, or actions that the user is asked to perform in sequence. method.

14. The digital collectible is a blockchain-secured digital collectible. The method of claim 13.

Images