Garment-based dynamic motion scoring

Abstract

A system for dynamic motion scoring includes an article of footwear or apparel and a processor in network wireless communication with the article of footwear or apparel. The article includes at least one accelerometer or inertial measurement unit operable to monitor spatial motion of at least a portion of the article of footwear or apparel and generate a data stream indicative of the monitored spatial motion. The processor is configured to receive the data stream from the article of footwear or apparel, identify at least one motion primitive from the received data stream, and determine an accuracy measure representative of a correspondence between the monitored spatial motion of the article of footwear or apparel and an ordered number of motion primitives.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 939,309, filed November 22, 2019, and U.S. Provisional Patent Application No. 63 / 032,689, filed May 31, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to a system for analyzing a user’s real-time movements using instruments provided in worn footwear or clothing items. Background Technology

[0004] In the fitness world and in motion-based social media challenges, users are often asked or challenged to perform a series of dynamic movements or choreographed dances in time with a beat or musical sample. While performing these challenges is sometimes a feat in itself, the real challenge lies in executing them completely, correctly, and smoothly from one movement to another. Without a coach present, it's difficult for individuals to truly understand how their performance compares to some idealized standard. Summary of the Invention

[0005] This disclosure relates to a system that can sense and process a user's real-time spatial movements using instruments provided in wearable footwear or clothing. These movements can then be compared to idealized standards, the movements of friends, or other users to better understand the accuracy and completeness of their movement trajectories. This mapping can be translated into an accuracy metric that can be shared with users, coaches, or the wider community via a distributed computing network.

[0006] In addition to simply scoring user actions, this system can also incorporate creative multimedia elements, where user actions can be manipulated to trigger the output of one or more audio samples or visual effects. In this way, users can perceive or express their actions through other sensory outputs.

[0007] Additional benefits and aspects of this disclosure are provided in the following disclosure. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of an embodiment of a system for motion-based media creation;

[0009] Figure 2 This is a schematic diagram of electronic motion-sensing footwear.

[0010] Figure 3 This is a schematic flowchart illustrating the operational methods of a system for creating media based on motion.

[0011] Figure 4 It is a schematic functional block diagram illustrating the operation of a system for motion-based media creation;

[0012] Figure 5 This is a schematic diagram of multiple wearable devices, each of which converts the motion it senses into audio / video output through a common set of rules;

[0013] Figure 6 This is a diagram illustrating multiple wearable devices grouped into pods. Each pod has converted the sensed motion of the relevant wearable device into audio / video output using a pod-specific set of rules.

[0014] Figure 7 This is a schematic diagram of multiple wearable devices, each of which converts the motion it senses into audio / video output through a different set of rules;

[0015] Figure 8 This is a diagram illustrating how a user utilizes this system to perform a motion-based challenge;

[0016] Figure 9 This is a schematic diagram of a user device displaying augmented reality visual effects generated by sensed motion from a connected wearable device;

[0017] Figure 10 This is a schematic diagram of a flash mob event, in which each participant has a wearable device with location-based access to audio / video output;

[0018] Figure 11 This is a schematic partial cross-sectional side view of a motion-activated light projector installed inside footwear.

[0019] Figure 12 It is a schematic perspective view of footwear with multiple light-emitting elements;

[0020] Figure 13 This is a schematic rear view of a garment item with multiple light-emitting elements;

[0021] Figure 14 This is a schematic partial cross-sectional view of a vibration transducer installed inside footwear.

[0022] Figure 15 This is a schematic side view of an inductive charger used to charge multiple wearable devices;

[0023] Figure 16 It is used as a party light. Figure 15 A schematic side view of an inductive charger;

[0024] Figure 17This is a schematic perspective view of an interactive retail kiosk;

[0025] Figure 18 For example, using Figure 17 A schematic flowchart illustrating the method for obtaining physical or electronic objects or codes from an interactive retail kiosk;

[0026] Figure 19 It is an illustration of a group of performance artists using connected wearable devices to influence performances in a hosted virtual world;

[0027] Figure 20 This is a schematic diagram of a wearable device used to enhance live video broadcasting. Detailed Implementation

[0028] The following discussion and accompanying figures disclose a system that uses directly sensed body motion to trigger the playback of one or more audio samples or visual effects. This technology aims to create a new form of expression in which the creation of electronic multimedia experiences is driven by the movements of dancers or athletes.

[0029] Beyond simply providing tools for building audio / video (A / V) experiences, some embodiments of this technology enable social collaboration among multiple users. For example, in some configurations, multiple users of the system can collaborate locally or networked to create collaborative A / V productions. In another example, multiple networked users can issue and / or respond to motion-based challenges from each other.

[0030] In collaborative environments, some embodiments of this technology can enable multiple members of a dance troupe or social network to collaboratively create A / V works, much like how a symphony is performed. Specifically, each user or a small group of users may have unique sound or visual effects associated with their movements. During a performance, the combined sound output generated by each member's movements can produce a performance-based A / V work, where the bodies of the various users become "instruments."

[0031] Regarding challenges in the context of social media, it has become increasingly popular for individuals to engage in online challenges with each other across various social media platforms. These challenges typically involve users performing one or more actions or dancing to a specific audio clip. An example of such a challenge involves a user filming themselves pouring ice water over their head and then issuing the same challenge to another user. Other challenges involve performing a specific or improvised dance sequence for a part of a song. In each case, users can record themselves performing the dance / actions and post the resulting video clips to online video hosting services. Examples of such hosting services include TikTok and Douyin, both operated by Beijing ByteDance Technology Co., Ltd., or YouTube, operated by YouTube, LLC, a subsidiary of Google LLC. As described below, this technology may be well-suited for similar “challenges.”

[0032] In some embodiments, the output expressed by the user may exist only in a separate medium and / or be consumed only by others (i.e., "viewers") located away from the user. For example, the user's movement can be used to trigger one or more audio and / or visual effects in a virtual environment, such as a video game. Alternatively, these effects can be presented in an augmented reality (AR) environment, where they can be overlaid on the natural perception of the real world. In this AR context, the effects can be transmitted to the user's device at the event site so that they can be superimposed on the user's real-world view—such as using AR display glasses. Alternatively, these effects can be superimposed on a captured video feed, such as a streaming video (Internet or television) broadcast, which can be viewed by a user device such as a mobile phone or television.

[0033] like Figure 1 The system 10 for motion-based media creation, schematically illustrated, may typically include electronic motion-sensing footwear or apparel 20 that communicates with a user console 30 via a network. For the purposes of this disclosure, the “electronic motion-sensing footwear or apparel 20” may generally be referred to as “wearable device 20”. Each wearable device 20 may be designed to resemble a conventional footwear or apparel item, although it may have additional electronic features to allow for the detection and transmission of motion data. Examples of suitable footwear items may include shoes, boots, sandals, shoes with cleats, etc. Similarly, examples of suitable apparel items may include shirts, jackets, trousers, shorts, socks, compression sleeves, gloves, hats, armbands / bracelets, etc.

[0034] In this technology, such as Figure 2As shown, the wearable device 20 may include motion sensing circuitry 22 and communication circuitry 24. Motion sensing circuitry 22 may include at least one accelerometer or inertial measurement unit (IMU) 26, operable to sense spatial motion of at least a portion of the wearable device 20 and generate a data output stream representing the sensed motion. In some embodiments, motion sensing circuitry 22 may also include a processor 28, memory, and / or any buffering or filtering circuitry, which may be necessary to prepare the data output stream for transmission and / or to convert the raw data output into a stream sequence of identified motion primitives (discussed in more detail below).

[0035] Communication circuitry 24, coupled to motion sensing circuitry 22, is configured to transmit a data stream to user console 30. Communication circuitry 24 may include one or more transceivers, antennas, and / or memories, which may be needed to assist in real-time or near-real-time data transmission. Such data transmission can be performed according to any suitable wireless standard; however, particularly suitable communication protocols include those based on any of the following standards or industry-recognized protocols: IEEE 802.11, 802.15, 1914.1, 1914.3; BLUETOOTH or BLUETOOTH LOW ENERGY (or other similar protocols / standards defined by the Bluetooth SIG); 4G LTE cellular, 5G, 5G NR, or similar wireless data communication protocols.

[0036] Refer again Figure 1 The user console 30, communicating with the wearable device 20, may include a computing device with operating software or firmware specifically designed to cause the computing device to perform as described below. The user console may include a processor 32, memory 34, and a user interface 36. Generally, the processor 32 used with the system 10 may be implemented as one or more digital computers, data processing devices, and / or digital signal processors (DSPs), which may have one or more microcontrollers or central processing units (CPUs), read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), high-speed clocks, analog-to-digital (A / D) circuitry, digital-to-analog (D / A) circuitry, input / output (I / O) circuitry, and / or signal conditioning and buffering electronics. Examples of suitable user consoles include smartphones, tablets, laptops, desktop computers, etc.

[0037] User interface 36 can be configured to provide the user with the ability to view and / or listen to available A / V effects, while also enabling the user to build a mapping table. In some embodiments, user interface 36 may include one or more displays 38 operable to output visual information to the user and / or one or more user input devices 40 operable to receive input from the user. Examples of user input devices 40 include touchscreens / digitizers, mice, keyboards and / or control panels with multiple knobs and / or buttons, cameras, gesture-based input devices, AR / VR virtual selection, etc.

[0038] Continue to refer to Figure 1 In addition to communicating with wearable device 20, in some embodiments, user console 30 may also communicate with one or more video sources 42, one or more audio sources 44, one or more audio output devices 46, one or more visual effects controllers 48, and / or a distributed computing network 50 (e.g., the Internet). Video sources 42 may include live video streams (e.g., from a digital camera), previously recorded video streams, digital storage devices storing one or more previously recorded videos, etc. Audio sources 44 may include one or more musical instruments, keyboards, synthesizers, data files, collections of audio samples, etc. In some embodiments, video sources 42 and / or audio sources 44 may be local to user console 30 or provided on a public local area network (LAN). However, in other embodiments, one or both of these sources 42, 44 may be located remotely from user console 30 and / or may be hosted by a computer accessible only via a network connection to the console.

[0039] One or more audio output devices 46 may include one or more speakers, amplifiers, headphones, or other devices operable to broadcast audible sound in response to received digital or analog audio signals. It is through these audio output devices 46 that the user console 30 outputs one or more audio samples in response to sensed motion. The visual effects controller 48 may include one or more devices operable to light one or more lights, initiate one or more light sequences, initiate one or more pyrotechnic effects, etc. In some embodiments, the visual effects controller 48 may reside on the wearable device 20, for example, to drive LED or fiber optic visual illumination. Examples of such implementations will be described in more detail below.

[0040] In operation, the user console 30 can be used to convert one or more sensed movements (i.e., the movements of the wearable device 20) into one or more auditory or visual effects. By linking various movements together, the user can be able to “play” a series of sound or visual effects. Figure 3The operation of the system 10 is schematically illustrated from the perspective of the user console 30. As shown, the method begins by establishing a motion mapping table (at 62) that associates sensed motion with desired audio / visual responses. This step is primarily an initialization step, and in some embodiments, it can be performed simply by loading a pre-established motion mapping table into memory.

[0041] Once a motion mapping table is established, console 30 can receive a data stream indicating the movement of the wearable device from footwear or clothing (at 64). User console 30 can continuously analyze this data stream to identify at least one motion primitive (at 66). As used herein, a "motion primitive" is a defined "block" of motion that represents discrete user actions. Examples of possible motion primitives may include sensed linear translation, arcuate translation, sinusoidal / periodic translation, rotation, acceleration, abrupt movement, and / or impact. In a general sense, motion primitives may include any combination of pre-programmed motion, user-defined motion, and / or automatically detected motion.

[0042] If one or more motion primitives are identified from the data stream (at 66), console 30 can trigger playback of an audio sample or visual effect previously associated with that motion primitive (at 68). In some embodiments, the audio sample or visual effect previously associated with the motion primitive can be a sequence of audio samples or visual effects, or a repeating sequence of audio samples or video effects. For example, upon detecting a user stomping their foot (e.g., a motion primitive characterized by a downward speed followed by a sudden deceleration), user console 30 can trigger playback of a single bass beat. Alternatively, it can trigger playback of multiple bass beats, and / or can trigger playback of a looping sequence of bass beats. In this way, the system provides the user with great flexibility to define what audio or video effect each movement (or movement sequence) can cause / initiate.

[0043] Figure 4 A functional block diagram illustrating the operation of the system is shown schematically. Each described module or block may include computer-executable code stored as software or firmware in memory, such that when executed, processor 32 can perform a specified function. In some embodiments, each block may also include any or all necessary hardware that may be required to perform the specified function or may be beneficially used. Examples of such hardware may include video or audio encoders / decoders, digital signal processors (DSPs), etc.

[0044] refer to Figure 4System operation typically includes two distinct modes: initialization 100 and media generation 102. During initialization 100, processor 32 may be responsible for building a motion mapping table 110, which associates sensed motion with desired audio / video responses. For this purpose, processor 32 may communicate with at least one of a pre-existing motion database 112, a video source 42, or a motion-tracking wearable device 20. Before building the motion mapping table 110, processor 32 may first compile a motion catalog 114 of available or anticipated motions that can be used for subsequent identification and playback.

[0045] In the simplest embodiment, the motion catalog 114 can be simply imported from a pre-existing motion database 112, which identifies typical or common movements in dance sequences, motions, activities, etc. If a generic motion database proves too cumbersome (i.e., too many unsuitable motion options) or insufficiently specific for the desired activity or dance, in some embodiments, the processor 32 may be able to build the motion catalog by parsing the movements of the connected wearable device 20 (i.e., via a received data stream) or by extracting motion information from the provided video source 42.

[0046] In embodiments where motion information is extracted from video, video source 42 may include, for example, live and locally captured video, pre-recorded video, and / or network or internet video feeds / streams. Video source 42 may be passed through object recognition and tracking module 116 to identify and estimate the three-dimensional motion of the depicted wearable device (or part of an individual's body). In one embodiment, object recognition and tracking module 116 may utilize image processing techniques such as boundary / edge detection, pattern recognition, and / or machine learning techniques to identify wearable device 20 and measure its movement relative to its environment or in a more object-centric coordinate system.

[0047] Once processor 32 has received the raw data stream from wearable device 20, or has identified the depicted motion of the wearable device from the video stream, it can transmit the raw motion through primitive detection module 118. In this module, processor 32 can examine the raw motion of one or more motion primitives or sequences of primitives. For each new primitive or sequence detected, processor 32 can categorize it in motion catalog 114 as a new general motion or motion type, a new specific motion, or a new motion sequence. A general motion or motion type can be, for example, translation (e.g., any translation) or impact. A specific motion can be, for example, a specific translation of wearable device 20 in a specific direction (e.g., translation of footwear in the inward direction or an impact of the left foot). Finally, a motion sequence can be, for example, multiple primitives arranged sequentially (e.g., a translation in the lateral direction followed by a translation in the inward direction).

[0048] Once a motion catalog 114 is established through direct import, active motion sensing, or video analysis and deconstruction, it can be presented to the user via an interactive user interface 36 along with a collection 120 of available audio samples and / or visual effects. The user interface 36 can receive user input 124 operable to link one or more categorized movements (i.e., movement type, specific movement, or movement sequence from the motion catalog 114) to one or more audio samples and / or visual effects from the collection 120 of available audio samples and / or visual effects. These relationships established between movements and audio samples and / or visual effects can then be stored in a correspondence table 110. In effect, the correspondence table 110 can be a translator that converts future movements into sound or light effects. In addition to establishing correspondences between movements and sound / light, the correspondence table 110 can also link one or more motion primitives to haptic responses, allowing viewers, if equipped with appropriate hardware for mechanical communication with their bodies, to feel a prescribed response following a particular motion primitive.

[0049] In some embodiments, the correspondence table 110 need not be a data structure completely separate from the motion catalog 114, but may simply include multiple pointers, each appended to a different corresponding motion entry within the motion catalog 114 and referencing a different effect. Furthermore, in some embodiments, the correspondence table may further include a set of rules that modify the prescribed output based on considerations such as the rhythm, beat, or timing of the movement. In this example, timing parameters may be used to change the pitch, tone, beat, or speed of the prescribed output (or the color, brightness, duration, or timing of the visual output).

[0050] Once the mapping table 110 is created and initialization 100 is complete, system 10 can then be set to media generation mode 102. In media generation mode 102, processor 32 can be operated to receive a data stream 130 from wearable device 20, which indicates real-time sensed motion of at least a portion of wearable device 20. The data stream 130 can be received via communication circuitry associated with processor 32 and can be available to the processor in real-time or near real-time. Based on the received data stream 130, processor 32 can analyze the motion using primitive detector 132 (which can be similar to or the same as primitive detection module 118 used during initialization 100). As described above, primitive detector 132 can examine the raw motion represented by data stream 130 and detect one or more motion primitives or primitive sequences.

[0051] To minimize processing time and thus improve the responsiveness of system 10, primitive detector 132 can be configured to search only in data stream 130 for motion primitives that have been previously defined in motion catalog 114 and / or assigned an associated audio sample and / or visual effect in correspondence table 110. Upon detection of a primitive, processor 32 can query correspondence table 110 and then, in response to at least one identified motion primitive, trigger or initiate playback 134 of the audio sample or visual effect.

[0052] In some embodiments, the set of available audio samples and / or visual effects 120 can be populated from a pre-existing library, which may be provided with the software or downloaded from a connected distributed computing network (e.g., the Internet). However, in one embodiment, a user may be able to populate or add to the set, such as by directly uploading one or more audio samples from a connected audio source 44 (e.g., a personal computer, digital music player, synthesizer, keyboard, etc.), or by recording one or more sounds or sound sequences generated by the system 10 due to user movement. More specifically, if a user creates a specific beat / rhythm / work by movement, they may be able to save the created work in the set 120 for future playback / triggering via a single discrete movement. In this way, different sounds / works can be layered to enhance the effect.

[0053] In one configuration, users have the ability to adjust the smoothness or autotuning between various audio / video effects. In this way, novice users can create pieces that sound or look well-made, even if their movements aren't 100% complete or timely. Similarly, wearable device 20 can be configured to predict motion primitives based on previous motion, or based on leading motion if the user is behind in their timing. Conversely, more advanced users can reduce smoothing / autotuning for more direct control over the output. In some embodiments, smoothing / autotuning can use machine learning and artificial intelligence techniques to blend audio / video elements together, which may rely on interspersed additional beats, lead-out notes, predicting subsequent motion based on earlier motion cues, etc.

[0054] In one configuration, user console 30 can be configured to acquire previously recorded / produced audio tracks / songs and divide them into multiple discrete segments (automatic segmentation). The nature, duration, and / or segmentation of the various segments can be customized by the user or even created individually (manual segmentation / segment modification). User console 30 can then automatically assign motion primitives to each segment, or can prompt the user to assign their own motion primitives to each segment. This can be analogous to choreographing dance moves for a selected song. The user (or group of users) can then initiate the playback of the song—segment by segment—simply by executing the choreographed movements in a timely manner. In one embodiment, the correspondence table 110 can be conditional based on other factors, rather than having an absolute correspondence table. For example, if a given motion primitive is executed between 00:00:00 and 00:00:10, or if it is executed as an initial movement, then the given motion primitive can initiate the playback of the first audio segment (of the song). However, if the same motion primitive is executed between 00:01:40 and 00:02:00, or if it is between the 10th and 20th identified primitives, or if it is after a different primitive has been identified or after the playback of the prescribed audio segment, then the same motion primitive can initiate the playback of the second audio segment. In another embodiment, the amount of smoothing / auto-tuning variation can also be applied to this sequential playback of musical segments, where segments can be mixed, or the final note can be extended to produce an output that flows from one segment to the next without appearing disjointed.

[0055] When choreographing for existing songs, the user console 30 may include local storage on which the songs are stored, or the console may communicate with, for example, an internet-based streaming audio source that the user may subscribe to individually.

[0056] In addition to simply outputting basic sounds or beats, or outputting snippets of pre-recorded tracks / songs, in some embodiments, the mapping table may include one or more motion primitives linked to audio action / control commands. For example, primitives can be used to initiate playback of a full song, initiate playback of the next song in the list, pause a song, rewind a song, change the beat of a song, change the pitch of a song, change the playback volume, fade in / out, etc. In this way, a user can act as a DJ or producer by using his / her movements to play back pre-recorded audio from a local or network source. Furthermore, audio action / control commands can be operated in conjunction with the display on the user's console. For example, primitives can be used to scroll through a list (e.g., to find a song / track).

[0057] For the purposes of any use case examples described herein, it should be understood that any audio playback may include discrete sounds, sound sets, pre-recorded sounds, snippets of audio tracks / songs stored on local storage, complete audio tracks / songs stored on local storage, snippets of audio tracks / songs extracted from an internet-based source (i.e., including songs that may be accessed via subscription-based login), or complete audio tracks / songs extracted from an internet-based source, etc.

[0058] As mentioned above, and usually as Figure 1 As shown, in some embodiments, system 10 may accommodate multiple user / wearable devices 20. In such a configuration, each wearable device 20 is configured to generate and wirelessly transmit a corresponding data stream 130 to user console 30, the data stream 130 indicating the monitored spatial movement of the device.

[0059] like Figure 5-7 As illustrated schematically, system 10 can be configured such that correspondence table 110 is globally set for all devices. Figure 5 ), for the equipment group being set ( Figure 6 ), or based on being set on a device-by-device basis ( Figure 7 If it is set globally, such as Figure 5 As shown, console 30 can apply the same set of rules / references to each connected wearable device (i.e., wearable devices 20a, 20b, and 20c). More specifically, data streams from each wearable device 20a, 20b, and 20c can be passed through their respective primitive detectors 132a, 132b, and 132c, and then referenced to a common correspondence table 110 before playback at 134. Although only a single table is used, different types of wearable devices may produce different results due to their different intended uses. For example, a watch is unlikely to withstand the same impacts as basketball shoes.

[0060] Figure 6 One embodiment is shown in which the processor 32 maintains different “pods” of wearable devices (i.e., wearable devices 20a and 20b, and wearable devices 20c and 20d), each pod having its own correspondence tables 110a and 110b (fed by respective primitive detectors 132a, 132b, 132c, and 132d). In this embodiment, multiple sets of wearable devices can be dedicated to their respective purposes. For example, a first set of wearable devices may primarily generate bass for output, while a second set of wearable devices may primarily generate piano-like tones or visual effects for output. During initialization, different wearable devices 20 can “subscribe” to specific pods, thereby acting as inputs to the A / V output linked to the corresponding pod.

[0061] at last, Figure 7 The use of multiple wearable devices is schematically illustrated, wherein each wearable device 20a, 20b, 20c is provided with its own corresponding tables 110a, 110b, 110c. In this multi-wearable device environment, the motion primitives of each wearable device can be separately assigned to the motion primitives of other wearable devices, so that similar wearable device-specific movements can each produce different outputs. For example, a lateral translation of a connected watch can result in the playback of a first sound (e.g., the impact of a cymbal), while a similar translation of a connected footwear item can result in the playback of a second sound (e.g., the impact of a bass drum).

[0062] Figure 5-7 Each embodiment shown provides a system in which multiple users can collaborate to create shared works. Figure 5 In the illustrated embodiment, each wearable device is a separate input to a common system. Due to the different nature and purpose of these devices, they may be able to create different motion primitives (e.g., footwear may be able to move differently than a watch), but ultimately, all motion primitives are input into the same database for output. A dance troupe might be interested in demonstrating how they move in sync (i.e., asynchronous movements result in similarly asynchronous beats). Figure 7 In this context, it resembles a small band composed of different instruments, with each wearable device capable of creating a separate sound / visual effect. In this way, multiple users can come together to collaborate on creations, each with their own tone and timbre. Finally, Figure 6 The illustrated embodiment can resemble a symphony orchestra, with groups of similar instruments; however, each group can have a unique tone and timbre. Ultimately, this technique can be used to create new artistic expressions for movement and is flexible enough to accommodate individual and collaborative efforts.

[0063] Refer again Figure 1 In other distributed multi-user scenarios, a user's local user console 30 can communicate with one or more remote user consoles 230 via a distributed computing network 50 for collaborative interaction or to create joint audio / visual experiences. Therefore, the physical distance between devices should not limit the boundaries of user creativity.

[0064] Furthermore, it should be noted that in each embodiment described herein, some or all of the user console may be physically integrated with wearable device 20. For example, in one configuration, remote user console 230 may not be a separate computing device, but rather a smart / connected wearable device 220, such as a smartwatch. In this embodiment, smart / connected wearable device 220 may include sufficient processing power and communication capabilities to transmit motion data to a distributed network and may even be able to communicate with one or more audio output devices 46 or visual effects controllers 48 via communication protocols such as Bluetooth. Similarly, smart / connected wearable device 220 may also serve as a user console 30 for one or more other wearable devices 20 that lack additional processing power.

[0065] Based on the concept of remote users collaborating on a distributed network, in some embodiments, the system can be used in a game-like context, where users can challenge each other to perform and / or create certain predetermined works of art, or reproduce certain dance sequences. For example, in one case, a sequence of movements (e.g., an ordered sequence of motion primitives) can be presented to the user on a display. The user can then attempt to replicate these movements in a timely manner, and if executed accurately, a pre-synthesized audio or visual output can be generated. Deviations in timing or completeness of the user's movements from the expected / displayed sequence will alter the auditory or visual output. Furthermore, in some embodiments, the processor 32 can be configured to determine an accuracy metric from the received data stream, representing the correspondence between the monitored spatial movements of the wearable device 20 and the ordered sequence of motion primitives. The accuracy metric can, for example, reflect deviations in timing, amplitude, and / or completeness of the user's movements relative to the presented sequence. In some embodiments, the accuracy metric may be the least squares value between reconstructed spatial curves. In other embodiments, the accuracy metric may include Frechet distance or other metrics that may represent deviations in ordered multidimensional points or curves. In still other embodiments, the accuracy metric may be a synthesis of multiple sub-accuracy metrics that describe different aspects of motion (e.g., integrity, acceleration, smoothness, transition, etc.). In some embodiments, the presented sequence may vary in difficulty or complexity based on the user's experience.

[0066] Figure 8An example of this distributed, challenge-based use is illustrated schematically. As shown, each wearable device 20 can communicate directly or indirectly with a distributed computing network 50, such as the Internet. The distributed network 50 may include one or more servers, computers, routers, switches, etc., which can facilitate network interconnection, data aggregation, and / or remote application hosting. In one embodiment of the user challenge, a first user 140 (or a group of users) can receive a sequence challenge 142 of movements / actions they wish to perform from a remote computing device 144 (which may be part of the distributed computing network 50) via their user console 30. Upon receiving the challenge 142, the first user 140 can attempt to reproduce the ordered sequence of movements or some variations thereof. Similar to the embodiments described above, each wearable device 20 is configured to generate a corresponding data stream 130 and wirelessly transmit it to the user console 30, the data stream 130 indicating the monitored spatial movements of the device. Based on the data stream 130, the user console 130 can generate associated audio / video output and can present the user with qualitative and / or quantitative scores 146 (i.e., accuracy metrics 146) of their performance. The score of 146 can be calculated by the user console 30 or the remote computing device 144, and can take into account factors such as timing deviations from a predefined beat, the integrity of motion with a predefined amplitude, and / or additional motion combined with the basic sequence as a means of modifying the original sequence.

[0067] After a work or challenge is completed, the user console 30 can transmit the audio or video capture 148 of the work and / or the accuracy metric / score 146 from the challenge to a remote computing device 144, where it can be hosted for viewing by one or more viewers 150 on a distributed network 50. Furthermore, the first user 140 can then issue a follow-up challenge 152 to a second user 154, either directly or via the remote computing device 144 and / or the distributed network 50.

[0068] In another example of cooperation / competition, this technology can be used to bring physically separated users together and / or gamify video-based workouts streamed over the internet, rather than issuing direct challenges to other users. For example, during an online / streamed Taekwondo class, each user watching or listening to a virtual lesson can be instructed to perform a series of repetitive movements or actions. The user's wearable device 20 (or a connected user console 30) can sense motion primitives associated with the user's corresponding movements and can combine this with the instructional video to overlay or display one or more visual effects on the user's display 38. In one embodiment, the color or nature of the visual effects displayed on the display 38 can change based on the similarity, completeness, or timing of the sensed primitives when compared to the expected or instructed movements. For example, if a user achieves a new personal best or exceeds a predetermined accuracy threshold, green stars may appear or colorful debris may fall from the top of the display. Conversely, if a significant deviation is identified, the display can provide the user with one or more visual stimuli. In one embodiment, a user's movements can be scored based on the accuracy or completeness of the movements when compared to movements of a coach or idealized reference. The accuracy score can be a moving average of accuracy over a predetermined time period and can be displayed on the user's monitor 38 for personal reference / goal setting. In cases where multiple users are distributed across the network and each user is watching the video, each user's accuracy score can be broadcast to a group (potentially anonymously), allowing each user to know their ranking. Similarly, a user's accuracy score can be displayed on monitor 38 alongside the scores of known influencers, professional athletes, or friends on the social network.

[0069] refer to Figure 9 In some embodiments, the spatial movement of one or more wearable devices 20 may be recorded by one or more user consoles 30, 230 and uploaded to a cloud-based distributed computing network 50 or a remote computing device 144. In some embodiments, a visual effects controller 48 may associate one or more visual parameters with the recorded spatial motion, such that the motion may be replayed or viewed by a device 200 with augmented reality capabilities. In such an example, the properties of the overlapping AR effects (color, trajectory awareness, visual effect sequence) may be the product of motion primitive types that occur when the spatial motion occurs. The device 200 with augmented reality capabilities may be a smartphone or AR glasses operable to overlay a graphic image onto the user's real-world view (i.e., perceived by the device), making the user perceive that the image exists at a specific real-world location.

[0070] like Figure 9As shown, in one embodiment, the spatial motion of one or more wearable devices can be represented by one or more persistent or time-decaying visual trajectories 202 superimposed on a displayed real-world image. The visual trajectory 202 can, for example, represent all or some of the motion in a basketball game, a single game, or a day. In some embodiments, the presence of different motion primitives can alter the visual effect, color, brightness, or persistence of the visual trajectory 202. For example, a rapid sprint on the court can be visualized as a elongated, thinner visual, while a sharp lateral cut can have a thicker, brighter, or more pronounced visual appearance. Although Figure 9 Visual trajectory 202 is simply illustrated, but in some embodiments, there may be corresponding wearable device-triggered audio effects that can be recorded and played back along with the video. Thus, motion primitives that result in significant trajectories, such as those likely caused by a dunk or a sharp cut, can also have associated audio effects.

[0071] In yet another embodiment, the visual trajectory can be scored using music stored in memory associated with the user's device 200, or music accessible to the user's device 200 via a streaming service / subscription. In this embodiment, the user's device can understand the timing and rhythm of the movement and the created visual output, and can select audio / music tracks with similar beats or rhythms. If the available audio does not slightly match the beat or timing of the movement, the user's device can be operated to select the closest possible audio and then modify the beat or rhythm to match.

[0072] The use of electronic motion-sensing clothing or footwear in this manner enables a new form of expression and creation that is impossible with other, more traditional electronic inputs. In sporting events, each sport and each athlete can create their own unique auditory or visual experiences, experiences that are distinctive to the athlete's style and performance. These experiences can be broadcast directly to the user (e.g., via speakers or lights within the facility) or via one or more handheld devices communicating with a user console paired with the athlete's wearable device. In a non-sporting sense, this technology enables performers (both professionals and those online at home) to have new digitally enhanced expressions where their own movements directly trigger the A / V experience.

[0073] Preset themes or styles can be used or applied in any of the examples cited above to introduce new sets of expressions while altering sound or visual expressions according to predefined rules. For example, an island / beach theme could change the available sounds to a sound set from the calypso theme, and / or change the available colors and / or visuals to colors and / or visuals within the blue / green / brown color range.

[0074] In one embodiment, such as Figure 10 As shown, access to or use of a specific mapping table (or related topic) can only be unlocked and / or become available when the user device / wearable device is present at a specific time and location. For example, members of a flash mob 300 can gain access to a predefined A / V topic upon arriving at the park or city square where the flash mob intends to perform. The user device / wearable device 20 can determine its location, for example, by using GPS, RF triangulation, Wi-Fi network identification, RF beacon 302, Bluetooth location tracking, etc. (commonly referred to as a "location sensing device"). When arriving at a predetermined location (or within a predetermined distance) at a predetermined time, any wearable device 20 that registers as part of the group with the connected server 304 can then be configured to begin recognizing the user's motion primitives and / or begin transmitting motion data to the distributed network 50. By requiring users to opt into joining the group, participation may be limited to those who knowingly wish to be part of the group, while those who happen to be in the correct location at the correct time will be excluded and / or prevented from joining the group unknowingly.

[0075] In the example of a flash mob, each wearable device 20 worn by a user can have a connection to a public distributed computing network (WAN, wireless LAN, etc.), either directly or via a smartphone, watch, or other connected device. The conversion from sensed motion primitives to triggered responses can occur locally on each wearable device 20, or more centrally on a networked server 304 / data aggregation device. Collective user output can then be converted into audio or visual output via a local A / V system 306 (e.g., for playback by local speakers 308 or visual effects devices), or it can be transmitted via a network to a third-party device 310, where it can be output as audio (via headphones or speakers) or visually displayed on a screen in augmented or mixed reality. Figure 9 The spirit shown is similar.

[0076] As described above, in some embodiments, one or more visual effects can be triggered in response to sensed motion primitives. Therefore, Figure 11-14Three embodiments of a wearable device 20 are schematically illustrated. The wearable device 20 includes one or more integrated light-emitting elements 400 that can be triggered in response to one or more detected motion / motion primitives to project visible light outward (e.g., under the guidance of a visual effects controller 48).

[0077] like Figure 11 As schematically illustrated, in one embodiment, the light-emitting element 400 may be a projector 402 operable to project light in a predetermined pattern onto an adjacent surface 404, such as the ground. As specifically shown, the projector may include a housing 406, a light source 408 such as a light-emitting diode (LED), one or more lenses 410 (e.g., one or more condenser lenses and / or objectives), and an optional slide or template 412. When used, the slide or template 412 enables the projector 402 to project images (e.g., logos, silhouettes, or graphics) onto the surface 404, rather than simply projecting a plain ball of light. In one configuration, the projector 402 may be integrated into the midsole 420 of the footwear article 422 and oriented such that it projects light through the outsole or the ground-facing outsole surface 424. In other embodiments, instead of projecting light through the outsole or the ground-facing outsole surface 424, the projector may be positioned / oriented to project an image onto a ground surface 404 adjacent to the footwear article 422. In such an embodiment, the projector 402 may be located on the sidewall of the midsole, or at least extend partially through the sidewall of the midsole.

[0078] In a configuration where the light-emitting element 400 emits light through the ground-facing outsole surface 424 of the footwear, the projected light is only visible to an observer when the wearer lifts their foot off the ground. To conserve power, in one configuration, the light-emitting element 400 can be controlled to illuminate only when the footwear detects its movement off the ground, or when it can be detected that the foot has moved a sufficient distance off the ground to make the broadcast light visible to an observer (i.e., the position can be derived from sensed acceleration data). In yet another embodiment, the light-emitting element 400 can be controlled to illuminate only when the wearable device 20 / motion sensing circuit 22 senses an upward acceleration above a certain threshold. This can indicate a jump or a jump of a specific magnitude and can further help ensure that the illumination has a minimum duration, or ensure that the projector 402 obtains an ideal focal length to project a sufficiently sharp image.

[0079] like Figure 12-13As shown, in some embodiments, the light-emitting element 400 may be a light-emitting panel 440 and / or an illumination design element 442, which may be disposed on or otherwise integrated with footwear or clothing articles (i.e., illumination articles 444). In some embodiments, the illumination article 444 may be the same as the wearable device 20 described above; however, in other embodiments, it may be different from the wearable device 20, although it may be in electrical communication with the wearable device 20 (directly or indirectly). For example, the wearable article 20 may be footwear, while the illumination article 444 may be gloves or a shirt. Article 444 may also include a power supply 446 and a controller 448, which is operable to regulate the power supplied from the power supply 446 to the panel 440 and / or the design element 442. In some embodiments, the controller 448 may be the visual effects controller 48 described above.

[0080] Figure 12 The illustration generally shows an illuminated article 444 as footwear article 460, which includes an upper 462 coupled to a sole structure 464. When the article 460 is worn, the sole structure 464 extends between the foot and the ground. In various embodiments, the sole structure 464 may include various cushioning components, including a foam polymer midsole, one or more integrated fluid-filled chambers (e.g., air bladders), a foam insole, etc. Furthermore, in some embodiments, it may include one or more recesses or cavities to accommodate a power source 446, a controller 448, and / or any other sensors or electronic components that the article 444 may use.

[0081] The upper 462 may include various means for receiving and covering the foot and for securing the article 444 to the foot. The upper 462 includes an opening 466 that provides an entry point for the foot into the cavity of the upper 462. Some embodiments may include fastening devices, including but not limited to: shoelaces, cables, straps, buttons, zippers, and any other means of fastening articles known in the art.

[0082] like Figure 12In general, as shown, in one embodiment, the outer wall of the upper 462 may include a light-emitting panel 440. The light-emitting panel 440 may include multiple different layers, at least one of which is an electroluminescent layer (or EL panel). Exemplary EL panel technologies that may be used include, but are not limited to, light-emitting capacitor (LEC) panels, powder phosphor-based electroluminescent panels, and thin-film electroluminescent materials. Additionally or alternatively, the light-emitting panel 440 may include one or more light-emitting diodes or other lighting elements that can illuminate areas of the panel when diffused. Further embodiments and details of an article 444 having a light-emitting panel 440 are described, for example, in U.S. Patent No. 10,182,608, which is incorporated herein by reference in its entirety.

[0083] In some embodiments, footwear article 460 may include one or more illuminated design elements 442. These design elements 442 may include spotlight features or discrete elements, as opposed to a more general illuminated area (e.g., luminous panel 440). However, in some embodiments, there may be overlap between the two (i.e., the design element may be a marker or highlight of the panel). In such cases, one or more illuminated design elements 442 may include luminous markers or decorations, such as those described in U.S. Patent No. 8,056,269, luminous strips as described in U.S. Patent No. 10,004,291, luminous cords as described in U.S. Patent No. 8,813,395, and luminous cavities or fluid-filled cushioning components as described in U.S. Patent No. 8,356,430 and U.S. Patent Publication No. 2009 / 0158622, each of which is incorporated herein by reference in its entirety.

[0084] Figure 12 Item 444 is depicted as a footwear item, while Figure 13Generally, item 444 is illustrated as clothing / clothing item 480, and more specifically as a shirt or jacket in a worn state. In other embodiments, clothing / clothing item 480 may additionally or alternatively include other types of shirts (long-sleeved, short-sleeved, vests, waistcoats), trousers, socks, tights, hats, gloves, outerwear, and / or any other desired type of clothing. As shown, clothing / clothing item 480 includes a lighting design element 442, which includes a lamp array 482, a plurality of embedded conductive traces 484, a power supply 446, and a controller 448. The lamp array 482 may include, for example, a plurality of discrete visible light sources (e.g., LEDs, OLEDs, electroluminescent materials, etc.) arranged, for example, in a single row or two or more rows of lights. The lamp array 482 may also include continuous light lines using, for example, fiber optic lights and / or optical fibers. The lamp array 482 is configured to be thin, flexible, and bendable to conform to the body curves when the clothing is worn. The light array 482 can be attached to the garment using a polymer layer such as thermoplastic polyurethane (TPU) or a silicone-based polymer. Further details regarding exemplary illuminateable garments / clothing are described in U.S. Patent Application Publications Nos. 2019 / 0059461 and 2019 / 0200690, both of which are incorporated herein by reference in their entirety.

[0085] In some configurations, Figure 12-13 The light-emitting element 400 can be used as Figure 5-7 One or more outputs of the user collaboration architecture shown. For example, in a collaborative visual or audio / visual work, multiple users can each wear one or more illuminated items 444, each illuminated item 444 having one or more light-emitting elements 400 disposed thereon. Different motion primitives or combinations of motion primitives can then trigger the illumination of different light-emitting elements 400, wherein color, brightness, duration, and / or illumination mode (e.g., steady, blinking, rapid blinking, scrolling, etc.) are either predefined in response to the motion primitives or functions of the amplitude of the primitives. For example, a sharper lateral cut can result in a brighter panel illumination compared to a less intense lateral cut (e.g., measured by the magnitude of lateral acceleration or the magnitude of lateral jerk (da / dt)).

[0086] In one example, such as Figure 13 The tight-fitting clothes or jacket shown can be used by stage performers who want to add visual impact to their performance. In another example, such as... Figure 12 and 13The illuminated clothing items shown can be used by runners or cyclists to signal to others nearby. For example, a runner wearing a smartwatch can raise their hand to signal to oncoming vehicles. Wearable device 20 can recognize the motion primitives associated with the raised hand, which can then trigger some of the light-emitting elements 400 to illuminate. In one configuration, different hand movements can result in different illumination patterns. For example, raising the hand upwards will illuminate the first set of lights, while raising the hand outwards will illuminate the second set of lights.

[0087] When using this technology to create audio / video performances, maintaining a beat without an external rhythm can be particularly challenging. More specifically, in such use cases, users may lack many external cues that can be used to set the tempo / timing (e.g., bass beats). This problem can be especially pronounced in collaborative ensemble productions, where in other cases, the beat of the song may be relied upon to synchronize the performer's beat and rhythm.

[0088] To help one or more users track beats and / or stay in sync with other collaborating individuals, wearable device 20 may include a vibration transducer 500 operable to transmit tactile sensations to the user's body. Figure 14 One embodiment of such a wearable device 20 is illustrated schematically. In this embodiment, a vibration transducer 500 is disposed within the sole structure 464 of footwear article 444. However, in other embodiments, the vibration transducer 500 may be disposed together with the upper 462 of footwear article 444, or disposed on / on clothing articles such as watches, armbands, shirts, trousers, shorts, gloves, hats, or any other clothing or accessories that come into contact with the user's body.

[0089] Vibration transducer 500 can operate under the guidance of processor 28 to deliver beats or other tactile timing signals to the user. In a multi-user environment, this timing signal can then be synchronized across each of the multiple users to help create timing for mastering the correct piece (e.g., during a flash). In one configuration, the vibration delivered to the user by vibration transducer 500 can be a switching vibration or a composite vibration, comprising a tactile waveform that switches according to a specified beat. The tactile waveform can be a vibration having a frequency range, for example, but not limited to, about 100 Hz to about 300 Hz, or about 140 Hz to about 210 Hz, or about 230 Hz to about 260 Hz. This vibration can be selected such that the user can most easily perceive such notification from vibration transducer 500 when it is provided.

[0090] The intended beat of a song or piece can be represented by the periodic transmission of a haptic waveform with a duty cycle of less than about 50%, or more preferably less than about 30%, or in the range of about 5% to about 25%. The beat can have a transmission frequency between about 0.1 Hz and about 3 Hz. More appropriately, if measured in beats per minute (BPM), the switching beats can be transmitted discretely from about 30 to about 160 times per minute to represent the haptic waveform. In some embodiments, it is not necessary to represent every beat of the piece. Instead, only certain synchronized beats can be represented (e.g., one of four consecutive beats or one of eight consecutive beats in the piece). Regardless of the specific frequency of the beats, the haptic waveform can represent short bursts of buzzing, and the beats are the timing of these short bursts of buzzing.

[0091] In some embodiments, instead of periodic vibrations transmitted by a vibration transducer, a similar beat can be transmitted by contracting or tightening a portion of the wearable device 20 (or footwear or clothing item in communication with the wearable device) around the user's body. To achieve contraction, the wearable device 20 (or other item) may include one or more tensioning mechanisms operable to tension one or more fibers, cords, shoelaces, closure mechanisms, or other fit adjustment aspects of the item tangentially around a portion of the wearer's body. In doing so, tension in the item can cause the item to contract or decrease in a radial dimension, which will exert a compressive force on the wearer. Items particularly suited to this compression include shoes (i.e., adaptive lace tensioners), compression armbands / clothing, and watches / bracelets.

[0092] In some embodiments, the tensioning mechanism may include a motor operable to wind / unwind tension fibers embedded within an article in response to an actuation signal. In other embodiments, the tensioning mechanism may include one or more linear actuators, fast-response active materials (e.g., piezoelectric actuators), or microelectromechanical systems (MEMS). The tensioning mechanism may be configured to periodically induce momentary compression on a user's foot / body in response to a provided switching beat. Further descriptions of tensioning mechanisms in the footwear context are provided in U.S. Patent Nos. 10,448,707, 2018 / 0199673, 2017 / 0265583, and / or U.S. Application Serial No. 16 / 694,306, each of which is incorporated herein by reference in its entirety.

[0093] Figure 15A charging device 600 that can be used with the wearable device 20 as described above is schematically shown. It will be understood that each wearable device 20 may include a battery that requires periodic recharging to maintain the device's ability to perform the functions described above. To provide waterproofing, the charging device 600 may be an inductive charging device including one or more induction coils 602. When each coil 602 is excited by an alternating current waveform and brought into the vicinity of the wearable device 20, the induction coil 602 is operable to magnetically induce a charging current within the wearable device 20. In the case of footwear, two inductive charging discs 604, 606 (i.e., one per shoe) may be required, wherein each disc 604, 606 includes an energizable induction coil 602. An example charging device is described in US 2017 / 0150773, which is incorporated herein by reference in its entirety.

[0094] like Figure 16 As illustrated schematically, in some embodiments, the charging device 600 may also include the ability to function as a disco ball or party light. More specifically, each charging disc 604, 606 may include multiple illuminating elements 608 (e.g., LEDs) operable to project light of varying intensities and / or colors. Furthermore, the charging device 600 may include mounting features 610, such as hooks or brackets that can support the charging device in a party-appropriate configuration. Mounting features 610 may be integrated with or otherwise attached to a central hub 612 from which each charging disc 604, 606 extends. The central hub 612 may include an energy storage device 614, a movement mechanism 616, and / or a light controller 618, while also providing suitable strain relief for any wires extending to the charging discs.

[0095] When used as a party light, the energy storage device 614 (e.g., a battery) can provide the power needed to illuminate the lighting element 608, as well as the power required to facilitate any external communication or drive any aspect of the motion mechanism 616. The energy storage device 614 can be charged simultaneously with the wearable device 20 when the charging device 600 is plugged into an AC power source. Conversely, when the charging device 600 is operated as a party light, the energy storage device 614 can consume energy.

[0096] The moving mechanism 616 can be configured to cause movement of the charging disks 604, 606 and / or the illuminating element 608 to provide a visual effect similar to a disco ball or other moving spotlight. The moving mechanism may include, for example, one or more motors, spring / rotor driven mechanisms, or articulated reflectors / lenses. The moving mechanism 616 can typically produce rotation or oscillation of at least a portion of the charging disk or illuminating element to alter the way light is projected. Figure 16As schematically shown, in a particular embodiment, the moving mechanism 616 can cause the central hub 612 to rotate relative to the mounting feature 610, which in turn can cause the charging disks 604, 606 to rotate about the central axis 620.

[0097] The lighting controller 618 can be responsible for illuminating one or more lighting elements 608 to produce a visual effect. In one configuration, the lighting controller 618 can simply illuminate the lighting elements 608 in a repeating pattern, which can be pre-programmed and / or user-controlled. For example, a predefined set of lighting patterns and sequences can be pre-programmed into the lighting controller at manufacturing time. The user can then select the desired pattern / sequence, for example, by clicking a button on the hub 612 or by selecting it via a connected wireless device (e.g., a remote control, smartphone, computer, etc.).

[0098] In another configuration, the light controller 618 can illuminate the lighting element 608 in response to audio sensed via an integrated or connected microphone. The light controller can, for example, emit pulsed light in response to a sensed low-frequency beat, or change the light pattern or color based on sensed pitch or frequency. In one configuration, the microphone can be integrated into a connected wireless device that can transmit raw sensed audio data, audio summary data (e.g., beat or pitch data), or program commands based on the sensed raw audio or audio summary data.

[0099] In another configuration, the light controller 618 can receive digital signals from the wearable device 20 and can illuminate the lighting element 608 based on one or more sensed motion primitives, as described above. Thus, the light controller 618 may include or be in communication with a visual effects controller 48, and the charging device 600 may be used as a means for displaying a visual representation of user movement (or supplementing the user's visual movement). In this way, the lighting elements 608 can synchronize their movement and / or flashing with the rhythmic movement of one or more people dancing near the light.

[0100] In some embodiments, instead of triggering playback of audio samples or visual effects in response to at least one identified motion primitive, the motion primitive can act as input to an external game. For example, in some embodiments, shoes can be used as input to a dance-based video game where a series of movements are displayed on a video display, and the user must do their best to replicate these movements to achieve a high score. In another embodiment, such as Figure 17As shown, the game can be a retail-centric game that requires a user to perform one or more movement sequences to obtain an unlock code, token, pass, license, or opportunity to acquire retail products or digital collectibles. In some embodiments, the retail products or digital collectibles can be limited-edition retail products, digital representations of retail products, cryptographically protected digital assets, or other items with limited or restricted availability, such as the cryptographically protected digital assets described, for example, in U.S. Patent Applications No. 15 / 931,764 and / or No. 16 / 707,720, the entire contents of which are incorporated herein by reference.

[0101] Figure 17 Overall, an interactive retail kiosk 700 is shown, which requires active user participation so that user 702 can obtain retail products 704, digital representations of retail products, unlock codes or part of unlock codes for obtaining physical or digital retail products, or digital recognition of achievements (e.g., virtual badges that can be shared on social media).

[0102] Figure 18 The diagram schematically illustrates a method 800 for acquiring physical or electronic objects or codes, for example, using... Figure 17 The interactive retail kiosk 700 is used to perform this process. As shown, the method can begin by recognizing the presence of a user 702 within a predefined physical proximity range of the kiosk 700 (at 802). In one configuration, the kiosk 700 may include a camera, an RFID reader, a pressure pad placed on the floor next to the kiosk 700, or a button that the user can press to alert the system of their approach. In another configuration, the user 702 may have a wireless computing device 706 that wirelessly digitally communicates with a server node 708 via a communication network 710. The wireless computing device 706 may have means for recognizing the user's location, such as a GPS receiver, a wireless communication component that can recognize proximity of the device to a wireless beacon 302 or a local area network, or a camera that can sense and recognize visual aspects or digital codes in the surrounding environment. Once the user's location is recognized, an application running on the device can send an indication of the user's relative or absolute location near the kiosk 700 to the server node 708, which can then initiate a game.

[0103] The wireless computing device 706 may also include the ability to authenticate user identity, for example by including an application that requires the user to enter a password, present identifying biometrics (e.g., fingerprint or facial recognition), or otherwise securely log in to the application. In step 804, the user's wireless computing device 706 can authenticate the user's identity and location as physically present within a predefined neighborhood or geofence of the booth 700.

[0104] Once a user is authenticated and identified in the vicinity of booth 700, server 708 can present the user with a specific challenge (at 806). In one configuration, the challenge may be presented to user 702 via a display 712 coupled to or communicating with booth 700. In another embodiment, the challenge may be presented to the user via a communication network 710 and a display mounted on their wireless computing device 706. Generally, the challenge may include a series of movements, motions, or actions that require the user to perform in sequence. For example, the challenge may include a dance sequence, a set of exercises such as stretching jumps, lunges, burpees, high knees, leg kicks, or a series of yoga poses.

[0105] Once the challenge is presented to the user at 806, the user at 702 can perform the requested physical activity, and the server node at 708 can determine that the challenge has been successfully completed (at 808). This determination can come from direct observation (e.g., via a camera or pressure pad set up in the booth at 700) or by receiving a digital indication that the challenge has been performed. In one configuration, the received digital indication can be provided by the wearable device at 20 as a transmitted sequence of motion primitives, or as a simple indication that the wearable device at 20 has detected a sequence of motion primitives that match or are very close to those expected during successful completion of the challenge.

[0106] After confirming that user 702 has successfully completed the challenge (at 808), pavilion 700 can then present the user with a code (at 810), which can be redeemed for retail products (at 812), or combined with other codes to redeem retail products (or restricted digital objects). Figure 17 As shown, in one embodiment, the kiosk 700 can provide the user 702 with a printed receipt 714 bearing a machine-readable code 716. In another embodiment, the kiosk 700 can digitally transmit the code 716 to the user's device 706 or to a user account that has been authenticated by the server 708 before the challenge was issued. Once transmitted, the code 716 can be stored in a digital locker / memory or a cold-store digital wallet, allowing the user to access the code in the future.

[0107] In one embodiment, kiosk 700 may be located in a retail store and may provide a code to the retailer or input it into an internet interface to redeem a retail product with a similar style or design to product 704 displayed in kiosk 700. In another embodiment, a user may need to acquire multiple codes, each from a different kiosk, to qualify for the product. In this case of multiple kiosks, the kiosks may be distributed across a geographic area and / or multiple retail stores, and the user may need to search for them, as described in U.S. Patent Publication 2019 / 0213619 and U.S. Patent Application No. 16 / 874,944, the entire contents of which are incorporated herein by reference. In this configuration, the user may be sent on a treasure hunt to accumulate the codes required to unlock the product. As discussed substantially in U.S. Patent Application No. 16 / 874,944, in one embodiment, this can be achieved by using turn-by-turn navigation built into the shoe (e.g., by selectively tightening the shoelaces or by selectively actuating). Figure 14 The tactile transducer shown guides the user to the next booth.

[0108] Although retail kiosk 700 is Figure 17 While shown as a physical product display, in other embodiments it can simply be a non-retail display, such as an oversized sign, symbol, or other marker that may or may not have direct interactive capabilities. In some embodiments, kiosk 700 can be a virtual kiosk (i.e., viewable in an augmented or mixed reality environment using a suitable wireless computing device), or it can be a virtual waypoint or location, such as a geofenced area.

[0109] While most of the publicly available content provided above focuses on creating audio and / or visual effects in a live environment, similar experiences can also be created in virtual worlds (e.g., such as...). Figure 19 (as shown) or for use in enhancing live events (e.g., as shown) Figure 20 (As shown).

[0110] refer to Figure 19It is becoming increasingly common for music artists 900 to broadcast live concert performances in online virtual environments. This virtual environment 902 can be hosted and / or maintained by a game server 904 connected to a distributed computing network 50 such as the Internet. Multiple users can interact with the virtual environment and with each other through unique, user-controlled avatars 906 within the environment 902. During gameplay, each avatar 906 can respond to corresponding movement / action commands made by a network-enabled computing device 908 controlled by a different corresponding user 910. Each user 908 can experience their avatar 906's first- or third-person perspective within the virtual environment 902 via a monitor associated with their respective network-enabled computing device 908. A popular example of such an environment is a game developed by Epic Games. exist.

[0111] exist Figure 19 In the illustrated embodiment, very similar to the embodiments described above, the user console 30 can receive a data stream from a wearable device 20 mounted on the body of the music performer 900. The user console 30 can rely on a similar motion mapping table 110 to trigger playback of audio samples or visual effects previously associated with the motion primitive. However, the key difference is that the triggered audio samples or visual effects only appear within the virtual environment 902. For example, a lateral swing of the performer's foot can cause the sky in the environment to change color, a stomp of the user's foot can trigger virtual fireworks, and a downward slam of the user's fist can result in a strong beat / kick playback, which may be accompanied by, for example, a visual shockwave emanating from the performer's avatar 912. Furthermore, instead of using the motion primitive solely to control sound or visual effects, motion primitives sensed by the performer's wearable device can also be used to control or alter some or all of the movements or actions of the performer's avatar 912 within the virtual environment 902.

[0112] Figure 19 The use of wearable devices 20 to control aspects of the virtual world, and Figure 20The system 950 shown is operable to enhance one or more aspects of a live video broadcast using sensed motion. In the context of a sporting event, a portion of an athlete's uniform may include a wearable device 20 operable to detect or extract one or more motion primitives from the wearer's movement / activity. If an athlete performs a predefined movement or a series of movements, the visual broadcast 952 of the sporting event (television, cable television, virtual reality, internet streaming, etc.) can be enhanced / altered with one or more visual effects 954 to emphasize or highlight the athlete's actions. For example, in a basketball game, if an athlete is about to dunk with a force or jump of a threshold magnitude, upon landing, the broadcast system 956 can overlay the video feed 958 with a graphic of an explosion emanating from the athlete themselves on the court surface. In another example, if a basketball player is about to perform a signature crossover move, this sequence of actions can cause the broadcast system 956 to graphically alter the perceived contours of the ground within the video feed 958—as if the athlete's stomping foot wrinkled the court surface. In doing so, this use of the technology injects aspects of a stylized or cartoonish video game (e.g., similar to NBA JAM produced by Electronic Arts Inc.) into live sports, while being triggered by the athlete's own directly sensed movements (rather than by a third-party visual observer).

[0113] In some embodiments, in addition to movement triggering, the generation of visual effects can also be conditional on external factors, such as the time of the match, recent scoring activity, or the relative position of other athletes on the court / field. Furthermore, in some embodiments, the presence, nature, or color of the effect can vary based on one or more preferences or attributes provided by the viewing user (i.e., via the user's respective network-enabled computing device 960). In such embodiments, the user can pre-select a favorite team or athlete, which can dynamically assign different sports correspondences to different athletes / teams. During video broadcast, the pre-selected favorite team or athlete can then be enhanced by a first set of sounds / graphics, while other athletes (especially opposing athletes / teams) can be enhanced by a second set of sounds / graphics. The difference in treatment between the designated favorite athlete / team and the opponent can have the effect of portraying the pre-selected favorite athlete as a hero and the opposing athlete / team as the villain. This can be achieved, for example, by using different colors, different graphics, and / or different sound effects.

[0114] Very similar to Enhanced Sports Broadcast 952, with visual effects ported to the Virtual Environment 902, such as... Figure 19As shown, this can vary depending on one or more attributes or preferences of the user in that world. For example, the clothing / skin worn by the user's avatar can unlock certain color schemes or visual effects. Similarly, the user's avatar's level or experience can unlock different visual or auditory effects. This can be achieved, for example, by applying completely different mappings, or by conditioned aspects of the mappings on certain criteria.

[0115] In addition to simply altering visual or auditory effects, the output of the mapping table can be transmitted to the user in the form of one or more tactile signals. Each tactile signal can instruct a tactile device on the user's body to provide a tactile response. These tactile responses can take the form of vibration or contraction of footwear or clothing items placed on the user and can be synchronized with one or more visual or auditory effects. In this way, the technology can appear through multiple senses to provide a more immersive user experience. In one embodiment, the tactile signals can further attempt to convey or impart one or more tactile sensations that can be similar to those experienced by athletes on site.

[0116] The various features and operating methods of the technology described herein are set forth in the following clauses:

[0117] Clause 1: A system for dynamic motion scoring, comprising: footwear or clothing article including at least one accelerometer or inertial measurement unit, the at least one accelerometer or inertial measurement unit being operable to monitor the spatial motion of at least a portion of the footwear or clothing article and generate a data stream indicating the monitored spatial motion; a processor wirelessly communicating with the footwear or clothing article via a network, the processor being configured to: receive the data stream from the footwear or clothing article; identify at least one motion primitive from the received data stream; compare the at least one identified motion primitive with a predetermined ordered number of motion primitives; determine an accuracy measure representing the correspondence between the monitored spatial motion of the footwear or clothing article and the ordered number of motion primitives; and display the accuracy measure to a user via a display device.

[0118] Clause 2: The system according to Clause 1, wherein the processor digitally communicates with a distributed computing network, and the processor is further configured to: receive a challenge from the distributed computing network, the challenge comprising an ordered list of motion primitives; and display the challenge to a user via a display device.

[0119] Clause 3: The system described in Clause 2, wherein the challenge is received from a second user on a distributed computing network.

[0120] Clause 4: The system according to Clause 3, wherein the processor is further configured to receive an accuracy metric from the second user and display the accuracy metric to the user via a display device.

[0121] Clause 5: The system according to Clause 1, wherein the processor digitally communicates with a distributed computing network; the processor is also configured to send accuracy metrics to servers on the distributed computing network.

[0122] Clause 6: The system according to Clause 1, wherein the ordered number of motion primitives form a choreographed dance.

[0123] Clause 7: The system according to Clause 6, wherein the footwear or clothing article further includes a haptic transducer operable to transmit tactile sensation to the user's body; and wherein the processor is further configured to cause the haptic transducer to transmit a beat or haptic timing signal to the user.

[0124] Clause 8: The system according to Clause 7, wherein the tactile transducer includes a motor operable to selectively and periodically tension at least one lacing or closure mechanism of the footwear article.

[0125] Clause 9: The system according to Clause 1, wherein the processor is further configured to trigger playback of an audio sample or visual effect in response to at least one identified motion primitive.

[0126] Clause 10: The system according to Clause 9, wherein the identified at least one motion primitive includes a first motion primitive and a second motion primitive, and the audio sample or visual effect is a first audio sample or a first visual effect, and is triggered in response to the first motion primitive; and

[0127] The processor is further configured to: trigger playback of a second audio sample or a second visual effect in response to a second identified motion primitive; and wherein the first audio sample or the first visual effect is different from the second audio sample or the second visual effect.

[0128] Clause 11: The system according to Clause 9 further includes a user input device and a display communicating with the processor, and wherein the processor is further configured to: maintain a library of multiple audio samples; associate audio samples selected from the multiple audio samples with predefined motion primitives based on input received from the user input device; match the identified motion primitives with the predefined motion primitives; and wherein triggering playback of audio samples or visual effects in response to the identified motion primitives includes outputting the selected audio sample in response to the matching of the identified motion primitives with the predefined motion primitives.

[0129] Clause 12: The system according to Clause 1, wherein the processor is further configured to modify the behavior or movement of an avatar in a virtual environment based on the received data stream.

[0130] Clause 13: A dynamic motion scoring method comprising: receiving a data stream from footwear or clothing articles, the received data stream representing the spatial motion of the articles; identifying a plurality of motion primitives from the received data stream; comparing the at least one identified motion primitive with a predetermined ordered number of motion primitives; determining an accuracy measure representing the correspondence between the monitored spatial motion of the footwear or clothing articles and the ordered number of motion primitives; and displaying the accuracy measure to a user via a display device.

[0131] Clause 14: The method according to Clause 13 further includes: receiving a challenge from a distributed computing network, the challenge comprising an ordered number of motion primitives; and displaying the challenge to a user via a display device.

[0132] Clause 15: The method described in Clause 14 further includes sending a challenge to a second user on the distributed network.

[0133] Clause 16: The method described in Clause 13 further includes sending an accuracy metric to a server on a distributed computing network.

[0134] Clause 17: The method described in Clause 13 further includes triggering playback of an audio sample or visual effect in response to each identified motion primitive.

[0135] Clause 18: The method described in Clause 17, wherein triggering the visual effect includes illuminating at least one light on the footwear or clothing item.

Claims

1. A method for controlling or enhancing a virtual environment or live video broadcast based on the real-time motion of sensed objects, the method comprising: Receive data streams indicating the spatial motion of the monitored footwear or clothing from accelerometers or inertial measurement units mounted on the footwear or clothing worn by the object; Identify at least one motion primitive from the received data stream; In response to identified motion primitives, visual effects are used to enhance virtual environments or live video broadcasts; Transmit a view of the virtual environment or a live video broadcast to the user for display on a display device; The transmission view of virtual environments or live video broadcasts includes visual effects; The visual effects include color changes in the virtual environment, virtual fireworks, or visual shockwaves emanating from the avatar of the object.

2. The method according to claim 1, wherein, The virtual environment or live video broadcast is a virtual environment; Furthermore, the virtual environment includes multiple avatars, including at least an object avatar and a user avatar.

3. The method according to claim 2, wherein, The visual effects include the movement or action of the object's avatar.

4. The method according to claim 1, wherein, The enhanced virtual environment or live video broadcast refers to the video broadcasting of sporting events.

5. The method according to claim 4, wherein, The object appears in the video broadcast.

6. The method according to claim 5, wherein, The visual effect is a graphic emitted from an object in the video broadcast.

7. The method according to claim 6, further comprising: Receive user preferences or attribute indications; and The presence, nature, or color of visual effects can be changed based on user preferences or attributes.

8. The method according to claim 4, further comprising: It receives instructions from one or more external triggers; and the presence of visual effects further depends on the presence of one or more external triggers.

9. The method according to claim 8, wherein, The one or more external triggers include: the time of the sporting event, the most recent scoring activity in the sporting event, or the relative position of other athletes in the sporting event.

10. The method according to claim 1, further comprising: Maintain a mapping table that includes multiple video effects, each video effect corresponding to a different predefined motion primitive or a series of predefined motion primitives; Match the identified motion primitives with one of the predefined motion primitives in the corresponding table; And among them, Enhancing virtual environments or live video broadcasts with visual effects includes: selecting visual effects from a mapping table that correspond to predefined motion primitives that match, and overlaying the selected visual effects onto the virtual environment or live video broadcast.

11. The method of claim 10, further comprising: Select a corresponding table from multiple corresponding tables based on the user's attributes or preferences.

Images