Airshow system with dynamic participation of unmanned aerial vehicles (UAVs) with distributed show system
A ground-based control system synchronizes UAVs with ground elements in aerial displays, addressing synchronization issues by dynamically adjusting show effects based on real-time UAV positions, enhancing display coherence and safety.
Patent Information
- Application Number
- JP2020147399
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-03
- Filing Date
- 2020-09-02
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2040-09-02
AI Technical Summary
Existing aerial display systems using unmanned aerial vehicles (UAVs) struggle with synchronization and coordination between pre-programmed flight paths and live human performances, leading to disjointed and arbitrary appearances of UAVs in shows.
A ground-based control system dynamically tracks and controls UAVs to synchronize their movements with ground-based show elements, using onboard and off-board show effects devices to create cohesive and interactive aerial displays.
The system ensures tight synchronization of UAVs with other show elements, enhancing the quality and coherence of aerial displays by adjusting effects in real-time based on actual UAV positions and performance, ensuring safety and dynamic interaction.
Smart Images

Figure 0007591372000001 
Figure 0007591372000002 
Figure 0007591372000003
Abstract
Description
[Technical field]
[0001] This disclosure relates generally to air shows or displays, and more particularly, this description relates to a novel system (and associated control method) configured to provide a dynamically coordinated show (e.g., coordination between airborne and ground devices) for the entertainment of patrons of venues such as theme parks, outdoor sports stadiums, etc. The system comprises a distributed show system and human actors, and unmanned aerial vehicles (UAVs) that are tracked, monitored, and controlled to provide dynamic show participation involving the distributed show system and the human actors. [Background technology]
[0002] There is a strong demand for new ways to provide shows or displays in the sky. These shows and displays may be provided, for example, to crowds visiting theme or amusement parks. In other cases, aerial displays or shows are provided to sports fans before or after sporting events, or even during half-time or other breaks.
[0003] Show and display designers are beginning to use unmanned aerial vehicles (UAVs), such as quadcopters, as part of the show or to play a part in the aerial component of the show. UAVs present many opportunities to creatively enhance theatrical show productions. However, UAVs typically operate as mostly independent systems that may fly in preprogrammed patterns. In particular, to date, UAVs have generally been controlled by executing a set of preprogrammed movements, such as performing a predetermined flight plan through space. UAVs used in a show are otherwise disconnected from the rhythm of the live performance of the rest of the production, such as performances by human actors, triggering of ground-based show or display systems, and / or the movement and operation of other UAVs in the show space.
[0004] Modern aerial display or show staging has increasingly complex human performances and show effects, and to date, synchronizing these elements with the pre-programmed flight path of a UAV has proven very difficult. The show staging may be simplified to making the UAV appear to hit a particular mark at the same time as other show elements, and this lack of tight synchronization can result in the appearance of the UAV within the aerial display or show being perceived as arbitrary and disruptive. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] U.S. Patent No. 8,876,571 [Patent Document 2] U.S. Pat. No. 10,424,231 [Patent Document 3] US Patent Application Publication No. 2014 / 0249693 [Patent Document 4] US Patent Application Publication No. 2019 / 0315486 Summary of the Invention
[0006] Briefly, an air show system is provided that includes an unmanned aerial vehicle (UAV), a show system on the UAV (including anything carried or moved by the UAV), human actors or show participants, a non-UAV or "ground" show system, and a global ground controller or system. A UAV in this description includes any unmanned object flying in the show space, such as a copter, drone, ballistic object (which may include a flying object launched into the air, or a robotic device that is dropped from an elevated platform or from a height to fly through the airspace of the show), etc. The global ground-based control system is configured to have hardware and software (e.g., a global controller module) to actively track the operation of the UAV during the show performance, and to respond and act on the UAV accordingly to make the UAV faithfully part of the larger show performance or experience. The new air show system overcomes the deficiencies by relying on having a UAV with on-board show effect elements that simply follow a pre-programmed flight plan. Instead, the new system achieves dynamic show participation of UAVs with a distributed show system that includes other UAVs with show effect elements, human actors on the ground or in the air, and non-UAV show systems that may be on the ground but that may launch or provide effects in the airspace where the UAVs fly during the show performance.
[0007] For example, a global ground-based control system may track a UAV having a show effect element (or an on-board show system having one, two or more show effect elements, and the show effect elements may each comprise a robotic subsystem or component) to determine whether the UAV properly hits a cue or mark with respect to its point and orientation in show space and with respect to timing (e.g., is the UAV's speed and direction of travel correct to provide proper time synchronization with the operation of other show systems and / or show effect elements). If "yes" and the cue / mark is hit, the ground-based control system may trigger a first effect by one or more show systems, but if "no" because the cue / mark has been missed by the tracked UAV, the ground-based control system may trigger a second effect (or a third effect, etc.) provided by one or more show systems (the same or different from those used to provide the first effect). During operation of the new air show system, the ground-based control system is aware of all UAVs used in the show space, each of which communicates with each other and with the ground for monitoring and show effect triggering operations by the ground-based control system.
[0008] More specifically, a system configured to achieve dynamic participation of unmanned aerial vehicles in a distributed air and ground show is provided. The system comprises a ground-based show control system that generates trigger signals to trigger show effects based on an ideal timeline defined for the distributed air show. Each UAV comprises a flight controller and, optionally in some cases, a flight propulsion mechanism operable to move the UAV in three dimensions in a show airspace. Additionally, the system comprises: (a) one or more show effect devices physically separate from the UAV and operable in response to the trigger signal to generate a first show effect; and (b) one or more show effect devices physically on or supported by the UAV and operable in response to the trigger signal to generate a second show effect. During system operation, the first show effect and the second show effect interact to generate a cumulative or complete show effect.
[0009] In some embodiments, the UAV further comprises an on-board show controller having computational resources to execute a script to generate show control signals for the one or more show effect devices physically on or supported by the UAV based on the show control signals. In such embodiments, the script may be or include a mini-scene including musical or lighting cues that are triggered and mixed based on the actual position of the UAV in the show space to maintain synchronization of the distributed air show across the timeline. The mini-scenes can be uniquely configured to end earlier or later on the timeline to suit the actual position of the UAV.
[0010] In the same or other embodiments, the ground-based show control system generates the trigger signal based on the timing of the UAV traveling to or through a location in the airspace relative to timing and marking positions defined in the timeline for the distributed airshow. The system may further include a tracking device coupled to the ground-based show control system that tracks a relative position of the UAV and the show effect device physically separate from the UAV, the relative position may be used by the ground-based show control system to select a time for issuing the trigger signal. The relative position may also be used by the ground-based show control system to generate the trigger signal that causes the one or more show effect devices physically separate from the UAV to provide lighting or projection toward the UAV as the UAV moves through the show airspace by means of the propulsion mechanism. In addition, the tracking device may determine real-time positions of additional UAVs in the show airspace, and the ground-based show control system may then automatically modify operation of one or more show effect devices to adjust lighting or projection to control lighting for one or more of the UAVs located between the UAV and the one or more show effect devices physically separate from the UAV.
[0011] In some implementations of the system, the trigger signal is generated at a time earlier or later than the timing defined in the timeline for the distributed air show. The one or more show effect devices physically on or supported by the UAV may provide light, video, or laser projection from the UAV to the one or more show effect devices physically separate from the UAV. In these or other cases, the one or more show effect devices physically separate from the UAV provide light, audio, video, or laser projection to or toward the one or more show effect devices physically on or supported by the UAV. The one or more show effect devices physically separate from the UAV may be configured to provide light, video, or laser projection to the UAV or to the one or more show effect devices physically on or supported by the UAV.
[0012] To enhance safe system operation, the ground-based show control system may be configured to generate the trigger signal only when the UAV is determined to be at a point safe for operation, such as pyrotechnic or projectile effects, of the one or more show effect devices physically separate from the UAV. The ground-based show control system may operate the one or more show effect devices physically separate from the UAV to highlight the UAV in the show airspace during an emergency landing of the UAV. The ground-based show control system may operate the one or more show effect devices physically separate from the UAV to highlight additional UAVs in the show airspace during the timeline of the distributed airshow. [Brief description of the drawings]
[0013] [Figure 1]FIG. 1 is a functional block diagram of an air show system dynamically employing unmanned aerial vehicles (UAVs) along with other distributed show effect elements according to the present description. [Diagram 2] FIG. 1 is a side perspective view of one exemplary implementation of an aerial display or show system in use with a UAV, with an on-board show system and a distributed show system in the show or display. [Diagram 3] FIG. 13 is a flow diagram of a control method for operating the show system to provide dynamic interaction of the UAV with show elements external to the UAV. [Figure 4] 1 is a flow diagram of a show control method that provides state-based control of a UAV through a show effect device. [Diagram 5] FIG. 13 is a flow diagram of a show system control method for safety-based control of show effect elements based on the tracked position of a UAV in the show airspace. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Briefly, an air show system is described that includes one to many multiple unmanned aerial vehicles (UAVs), each having an on-board show assembly with one or more show effect elements. The air show system also includes one to many distributed ground-based show systems, each having the show effect elements. The air show system is configured to track the UAVs during a show performance to coordinate and / or synchronize operation of the UAVs with operation of the ground-based show systems (which are typically spaced some distance apart from the UAVs), and to enable dynamic show participation of the UAVs and their show effect elements, along with the distributed ground-based show systems and the distributed ground-based control system show effect elements, to achieve a highly orchestrated show. In some embodiments, one or more of the ground-based show systems provide outputs or effects that interact or combine with outputs or effects of one or more of the UAV's on-board show systems to achieve a complete show effect or cumulative effect that requires careful synchronization of the on-board and distributed show systems when the UAV hits a UAV mark / cue defined in the show flight plan and when the UAV goes off course or off schedule in the show airspace.
[0015] The new airshow system is particularly well suited to provide a show or display in the sky or in space above a group of spectators or observers below (e.g., on the ground). The aircraft or airframe may take the form of nearly any UAV, such as a tricopter, quadcopter, or other similar device configured to lift and / or carry heavy loads, while other embodiments may use UAVs in the form of airships, planes, and / or ballistic objects (such as launched or dropped animatronics or robots), such that "UAV" is intended to be interpreted very broadly. The show system or an assembly having one or more show effect elements of the show system is "on-board" in the sense that it may be physically on or within the body or structural frame of the UAV, or may be tethered to or coupled with the UAV so as to be suspended below the UAV or drone. The UAV is thus configured to support or carry a portion of the overall show system, and an on-board or local show controller is provided to the UAV to control the operation of the on-board show system / assembly components. Control may be provided by generating show control signals for the components of the on-board show system dynamically based on the output of the UAV controller as appropriate for a given show stored in local memory (e.g., providing data such as the UAV's speed, current location, etc.). Additionally, synchronization with other show systems may be provided through communication with a ground-based show control system that tracks the location and movement of each UAV in the show airspace (e.g., with a tracking device or system) and drives or provides a central or main show controller to provide timing and trigger signals to each show system on each UAV and to distributed ground-based show systems.
[0016] FIG. 1 shows a functional block diagram of an air show or display system 100 that employs the novel control methodology described herein. The system 100 comprises a UAV (or drone) 120 that physically supports, as indicated by dashed lines 151, an airborne show system or assembly 150 to provide one or more air show elements via show effect devices 168, such as a flying projection screen, a flying illumination system, a flying projection launcher, or the like. In other words, the UAV 120 is configured to support the show assembly 150 and to move the show assembly 150 (such as in three dimensions or in any direction indicated by arrow 125) in space 104 generally above ground 102, where the show assembly 150 or effects produced by the show assembly 150 can be viewed by an audience (not shown, but understood) located on ground 102. The show assembly 150 or show effect devices 168 of the show assembly 150 may be supported by a connection (indicated by dashed lines) to the UAV landing gear / body 122.
[0017] The UAV (or UAVs) 120 may take many forms to carry and position the on-board show assembly 150. In general, the UAV 120 may be any unmanned aerial vehicle or object (such as ballistic objects and propeller-based vehicles such as planar drones, such as airships / balloons, and helicopter-type drones using one or more propellers such as tricopters, quadcopters, and aircraft with 5, 6, or more propellers), including vehicles that can move or be propelled through space 104 by a propulsion mechanism 124, with propulsion for the 3D movement 125 of the landing gear / body 122 (and supported show effect devices 168) being provided by the flight propulsion mechanism (e.g., a combination of motors and propellers) 124. The UAV 120 is often chosen because of the UAV's ability to support its own weight while also supporting the weight (and movement in flight) of the on-board show assembly 150. The UAV 120 further comprises a power source 128 (such as one or more batteries) to provide power for the operation of the mechanisms 124 and other power-requiring components of the UAV 120. Sensors 126 are provided to sense the operation of the UAV 120, including the propulsion mechanisms 124, and to determine operational parameters such as the roll, pitch and yaw of the landing gear / body 122, the speed (and in some cases the direction) of the landing gear / body 122, and the position / location of the landing gear / body 122 in space 104.
[0018] The UAV 120 also comprises a processor 130 which manages communications (typically wireless) with a ground-based show control system 170, as indicated by arrow 171, and with other UAVs 190 of the system 100 (which may have their own show effects assemblies 193), as indicated by arrow 192. Input / output (I / O) devices 132 are provided for this purpose and may include wireless transceivers and the like, as is well known in the UAV industry. The UAV 120 comprises a memory / data storage 136 managed by the processor 130, and a flight controller 134 (e.g., hardware and software) also managed (or code executed) by the processor 130. The flight controller 134 may process and execute a flight plan 138 stored in memory for a particular show / display to be performed during operation of the system 100, and accordingly generate flight control signals 135 that are provided or communicated to the flight propulsion mechanisms 124 to operate the flight propulsion mechanisms 124 to move (125) the UAV 110 from one position to the next at a desired speed along a flight path, as indicated by arrow 125. The signals 135 may be generated based on output from the sensors 126, such as based on a current speed and heading 142 and / or a current point / position 144 in space 104 relative to a desired show mark / point and timing defined in the show flight plan 138. Additionally or alternatively, the signals 135 may be generated by the flight controller 134 based on received flight control signals 140 from a ground-based show control system 170, as indicated with communication 171, and / or based on inputs from the on-board show assembly 150 and a control software module / show controller 156 of the on-board show assembly 150.
[0019] The show assembly 150 may include machine interfaces, sensor interfaces, and / or show interfaces 158 to enable the on-board show assembly 150 (or its show controller 156) to communicatively interface with the propulsion mechanisms 124, the sensors 126, and / or the flight controller 134. As can be seen from this description and FIG. 1, the system 100 includes a UAV having a landing gear 122 that houses or supports a flight propulsion mechanism 124 capable of lift and / or movement in three dimensions, as indicated by arrow 125. The flight controller 134 is provided to the UAV 120 and coupled to the flight propulsion mechanism 124 to generate and / or receive flight control signals 135 and 140. Mechanisms 124 and / or sensors 126 operate to indicate an absolute or relative point 144 in space 104, and a flight controller 134 causes flight propulsion mechanisms 124 to move (125) the UAV landing gear 122 to a position indicated by flight control signals 135 (and signals 140), which are often defined in a show flight plan 138.
[0020] The entertainment platform 110 comprises an on-board show assembly 150 supported by the UAV 120 for movement (125) by the UAV 120. The show assembly 150 comprises a processor 152 that manages the operation of I / O devices 154 to communicate with non-on-board devices, such as communications 171 with a ground-based show control system 170, and communications 192 between other UAVs 190 in the space 104 (e.g., other UAVs having show effect devices 168 and a distributed show assembly 151 and a show assembly 193 for providing a show through coordinated operation with the show effect devices 157 of the distributed show assembly 151). The processor 152 also executes code / instructions or runs software in local memory 160 to provide the functionality of an on-board show controller or control software module 156.
[0021] In particular, the on-board show controller 156 serves to process a script or show plan 161 to generate a set of show control signals 164 that are communicated to one or more show effect devices 168 to operate actuators 169 of the show effect devices 168 to produce particular show effects. The assembly 150 includes interfaces 158 to the mechanisms 124, to the sensors 126, and / or to the flight controller 134 to enable it to receive vehicle control state information 165 (e.g., flight speed, landing gear orientation (yaw, pitch, and roll), etc.), current speed and heading 142, and current position 144 for the UAV 120. The show controller 156 may generate timing / trigger signals 162 and position signals 163 from a script / show plan 161 or receive them from the flight controller 134 or a ground-based show control system 170 to achieve dynamic participation, and the timing / trigger signals 162 and position signals 163 may be used independently or in combination to produce show control signals 164.
[0022] The show plan 161 may include two or more scripts (or branches for new show segments) that may be dynamically selected by the show controller 156 based on the timing signals 162, the position signals 163, and / or the vehicle control state information 165. In this manner, the show effect devices 168 may operate to provide desired show effects at predetermined or dynamically selected timings and / or positions in space 104 of the UAV 110. In some preferred embodiments, the dynamic selection is accomplished based on communication 171 from a ground-based show control system 170, which coordinates the operation of the on-board show assembly 150 with the operation of the distributed show assembly 151 using the ground-based show effect devices 157. The show controller 156 or main show control module 176 may also generate alternative scripts in real time for one or more of the show effect devices 168, 157, and 193. A given script / branch of the show or a real-time generated script / branch of the show (e.g., for real-time substitution) may be selected or created by the show controller 156, or more typically by the main show controller 176, based on real-time data such as the UAV's current position 144, current speed and heading 142, and / or the current orientation of the UAV landing gear 122 (and thus of the show effect devices 168), as may be provided to vehicle control state information 165 from sensors 126. In some cases, the real-time data used to select "B", "C", or other show script or branch in the show plan 161 may include the position of a nearby UAV 190 in the space 104 to improve the quality of performance of the show plan 161 (or show plan 186).
[0023] All or a subset of the show effects devices 168 may be provided on the UAV landing gear 122 or may be tethered or coupled to the landing gear 122, such as suspended below or behind the UAV 120 during flight. The show effects devices 168 each include one or more actuators 169 that, in response to receiving show control signals 164, operate the corresponding show effects device 168 to operate in one or more predetermined ways (e.g., to illuminate with a particular brightness range and / or color, to play a soundtrack at a particular volume, to project a video or still image, to emit a particular projectile or set of projectiles, etc.). The show effects devices 169 may take a variety of forms to implement the entertainment platform 110, such as an illumination system (e.g., one to many white or colored LEDs, lasers, black lights, etc.), a sound system having a playback device and one or more speakers, a pyrotechnic system for creating one or more pyrotechnic effects, and / or a projection system (e.g., for releasing or ejecting projectiles such as confetti, streamers, water droplets or streams, coupons, souvenirs, and other objects).
[0024] In some implementations, the on-board show assembly 150 is implemented distributed, with some components on or within the UAV landing gear / body 122 (such as the processor 152, memory 160, and controller 156, such as the show effect devices 168, and any other combination) and some components tethered or coupled to the landing gear / body 122 but spaced some distance apart (e.g., a screen or object suspended below the UAV landing gear 122). The on-board show system or assembly 150 comprises computational resources such as the processor 152 and controller 156 that communicate with each other, such as through an interface 158. An on-board show controller 156 having a processor 152 (and in some cases a memory 160) may be located entirely within or on the landing gear / body 122, or on / within a show effect device 168 or structural component (not shown) of the assembly 150, and may receive power from a power source 128, and may receive data from the host UAV landing gear 122 via an interface 158 (e.g., through a show interface) from the flight controller 134 (or directly from the propulsion 124 and / or sensors 126). The on-board show controller 156 may be connected to the various on-board show effect devices 168 to directly control and sequence the operation of the on-board show effect devices 168.
[0025] In some embodiments of the system 100, the on-board show controller 156 receives timing and trigger signals 171 from a ground-based show control system 170. To this end, the ground-based show control system 170 comprises I / O devices 174 for communicating with the I / O devices 154 of the on-board show assembly 150. The I / O devices 174 may be managed by a processor 172, which also executes code and / or runs software to provide the functionality of the main / central show control module or show controller 176. The show control system 170 further comprises a memory / data storage 180 that stores, for each UAV 120, 190 in the show system fleet, a record or file 182 that stores data specific to that UAV 120 or 190, such as the UAV's 120 or 190 current speed 183 and position / point 184 in space, as well as other operational data 185 (e.g., the orientation, operational status, etc. of its show effect devices 168). The memory 180 may store, for each UAV 120, 190, a show plan 186 that defines the flight path and a show script for each show system / assembly 150, 193. To this end, the ground-based show control system 170 is further shown to comprise a tracking device 178 that is operative to track or determine the current position 184 and speed 183 of each UAV 150 and 190 in the airspace 104 during a show performance. Such tracking may be performed in part by processing communications 171, 191 from the UAVs 120, 190 and / or through the use of software and / or hardware provided by a tracking device 178 (well known in the UAV / drone industry and not described in further detail herein).
[0026] The system 100 further comprises one to many distributed show assemblies 151, often ground based or on or supported by the ground 102, that are spaced apart from one another and from show effect devices 168 in the airspace 104 during show performance to provide distributed show effects. Each assembly 151 comprises a controller 153 operable to communicate with a ground based show control system 170 to receive timing and trigger signals, as indicated by arrow 155. In response, the controller 153 can operate one or more of the show effect devices 157 (e.g., based on a triggered script or a single or set of control signals) to provide a show effect (e.g., to trigger a series of laser lights, to project an image into the space 104, to operate a spotlight to provide light to the space 104, to launch a projectile, etc.).
[0027] In many show designs, the show effects device 168 (or in the assembly 193) operates to provide an output or to provide a first show effect in conjunction with the operation of one or more of the ground-based show effects devices 157, synchronized in time and position (and sometimes orientation) (sequentially, one or the other before the other, or simultaneously together at a particular time along the show timeline) to provide a second show effect 195, which, when combined, provides a third or cumulative show effect 196 in the airspace 104. For example, the show effects device 168 may operate to stretch or present a screen or surface, while the show effects device 157 may provide a projector or illumination device to project an image or light onto the screen or surface. In another example, the show effects device may launch projectiles or fireworks into the space 104 at the same time or sequentially before or after the show effects device 157 provides a synchronized fireworks or pyrotechnic show effect. Synchronization of the two effects 194 and 195 enables the interaction of the two effects 194 and 195 to create a new cumulative effect 196, which is of high quality due to the operation of the ground-based show control system 170 to provide dynamic interaction with the distributed show effect devices 157 of the show assembly 150 on the UAV 120.
[0028] During operation of the system 100, the on-board show controller 156 may receive timing and trigger signals 171 (also stored in on-board memory 162, as shown) from a ground-based show control system 170, for example, based on processing the output of the tracking device 178 in addition to the show plan 186. The system 170 operates independently of the safety-critical flight controller 134 of the UAV 120, but in coordination with the real-time position 144 (and signals 163) of the UAV landing gear 122. The geographic position (and position signals 163) can be used to activate show effects by operation of show effect devices 168 via control signals 164 based on pre-programmed conditions (e.g., entering and leaving a particular geographic area at a particular time to hit a mark / cue in the show) in real-time synchronization with the flight path of the UAV. In some cases, the on-board show controller 156 may use vehicle control state information 165 to improve safe operation and recovery of the platform 110. This may include the controller 156 using the nearby location of participating UAVs 190 to suppress show actions (e.g., operation of show effects devices 168) that may cause hazards (e.g., not activating on-board pyrotechnic effects when in close proximity to another UAV or platform 190).
[0029] 2 is a side perspective view of one exemplary implementation of an aerial display or show system 200 during use of a UAV with an on-board show system and a distributed show system in a show or display. In particular, a single UAV 220 to facilitate understanding of the single UAV 220 from FIG. 1, as well as the preceding description that additional UAVs with the same or different propulsion mechanisms and / or the same or different show assemblies / payloads are often provided in the system 200. The UAV 220 operates to move in three dimensions as indicated by arrow 221 in the airspace 204 by the propulsion mechanism of the UAV 220, which as shown comprises four rotors / propellers. The UAV 220 has an on-board flight controller that operates the propulsion mechanism to attempt to follow a pre-programmed flight plan, with updates and changes based on conditions and based on additional control signals from a ground-based control system (not shown, but understood from FIG. 1). The ground-based control system also communicates with UAV 220 and / or uses a tracking device to determine and track the 3D location of UAV 220 in the airspace 204 during the show, to determine when UAV 220 is hitting its mark / cue, and to otherwise modify the operation of UAV 220's on-board show system and / or the ground-based show system to provide dynamic interaction between those show components.
[0030] In this regard, the system 200 comprises an on-board show system or assembly 250 tethered to (or supported under) the structural frame / body of the UAV 220 for moving (221) with the UAV 220 around the airspace 204. In this example, the show assembly 250 provides a set of show effect devices or elements. These include a projection screen / display surface 252, which in this example is shaped as a sailboat, and a show assembly 260 on the ground 202 that may be operative to project an image or output light 261 onto the screen / surface 252 when the UAV 220 is located at a particular location with a desired orientation at a preset time (or a time selected by a ground-based control system), such that these two show elements interact to achieve a desired cumulative show effect.
[0031] Additionally, system 200 includes a ground-based show assembly 262 in the form of a projection screen / display object or surface 262, and an on-board show assembly 250 including projectors / light elements (e.g., spotlights) 254 that are triggered to project or output images / light 255 onto the screen / surface of the ground show assembly 262 (e.g., show elements 262 and show elements 254 interact to form a cumulative light or show effect) when the UAV 220 moves the show assembly 250 to a 3D point in space 204 at a predetermined (or later determined by the main show controller) location in the show timeline. Still further, the system 200 includes ground-based show effect elements 264 that are triggered by the ground-based control system in synchronization with the on-board show effect elements 256 to output fireworks or pyrotechnics 265, which also output fireworks or pyrotechnics 257, whereby the fireworks or pyrotechnics 265 and the pyrotechnics 257 interact or intersect in a portion of the airspace 204 to create a cumulative or new show effect 267. Also, actors (human or robotic) 268 perform show elements, and the ground-based show controller triggers the operation of show effect devices 258 in the show assembly 250 when the UAV 220 is in a particular 3D position at a selected time to provide the performance or show element in synchronization with the performance by the actors 268. For example, the devices 258 may be illumination systems (which may include white or colored spotlights) to provide light 259 that is directed toward a stage or space where the actors 268 are performing.
[0032] As can be seen from Figures 1 and 2, the airshow system may be usefully configured to provide dynamic participation of UAVs / airframes and show effect devices carrying distributed show effect devices to provide a planned airshow or display. The distributed show effect devices may be on or supported by other UAVs in the airspace and / or on the ground below or near the airspace. The UAVs / airframes have flight propulsion mechanisms that can lift or move the UAV landing gear in three dimensions in the airspace. The airshow system comprises ground-based and on-airframe show control systems for generating and / or receiving timing and position signals, which also comprise computational resources configured to execute scripts operable to generate show control signals based on the timing and position signals.
[0033] The airshow system comprises a first set of one or more show effect devices (e.g., light elements, audio devices, projectors, fog machines, pyrotechnic assemblies, etc.) physically coupled to the airframe / UAV. Each is designed to be triggerable in response to receiving a show control signal. The airshow system further comprises a second set of one or more show effect devices physically separated from the UAV / airframe, either on the ground or on another airframe / UAV, each of which is also designed to be triggerable in response to receiving a show control signal. During operation of the airshow system, one or more of the airframe-based or on-board show effect devices interact with one or more of the off-airframe show effect devices to provide a complete or cumulative show effect. The device interactions may include light, video, and / or laser projection from the airframe-based show effect devices to the off-airframe show effect devices (or from the off-airframe show effect devices to the airframe-based show effect devices). The device interactions may include projectiles traveling between an on-board show effects device to an off-board show effects device (or an off-board show effects device to an on-board show effects device).
[0034] 3 illustrates a control method 300 for a show system to provide an air show with dynamic interaction of tracked UAVs with other off-UAV show components. Method 300 begins at 310, such as with providing a ground-based show control system (e.g., system 170 of FIG. 1) and defining a show plan having a timeline providing timing for multiple show effects and how such show effects are achieved. As described above, some of the effects are achieved by interaction between the UAV and / or its on-board show system (with its show effect elements / devices) and distributed show effect devices in or near the show space (non-on-board the tracked UAV) to achieve a complete or cumulative show effect. Step 310 also includes providing one or more UAVs with (or without) an on-board show system and a distributed show system designed to provide show effects for the show plan.
[0035] Method 300 continues at 316 with retrieving or accessing a timeline for the show defined by the show plan. In this way, the ground-based show control system knows the timing of the show effects and the show effect device is intended to operate to achieve those show effects and the required positions for the particular UAV at those show effect times (e.g., marks / cues for each UAV performing the show plan). Method 300 continues at 320 with the ground-based show control system operating the tracking device to track a first one of the show UAVs to determine the current operating parameters of the first UAV, including speed, position, and orientation (and in some cases, the operating status of the show effect elements of the first UAV). The tracking device may also operate at 330 to determine the location of show effect elements (e.g., off-board show effect elements) that are not on the first tracked UAV. In some cases, these show elements may be provided at fixed points on the ground, whose locations can simply be retrieved from memory / data storage accessible by the show control system, in other cases, these show elements may be movable during the show, such as on the ground or on / by another UAV flying in the show's airspace, such that the locations of other show effect elements may change over the show's timeline.
[0036] At 340, method 300 continues with a control module (or tracking device) of a ground-based show control system operating to calculate the relative position of the first tracked UAV and / or its show element and the show element tracked in step 330. Then, at 350, the show control system operates to determine whether it is time to trigger the next show effect involving the first tracked UAV based on the retrieved show timeline. If not, method 300 continues at 320 with tracking the first UAV in the show airspace. If so (or that time is fast approaching), method 300 continues at 360 with determining whether the other show element tracked in step 330 is a cooperating UAV. If so, the show control system operates at 370 to modify the operating parameters of the show element of the first tracked UAV or of the cooperating / other UAV, when necessary, to achieve the desired show effect. The show element is then triggered to provide the show effect defined at this time in the show timeline, at 380. The method 300 may then end, such as at the end of the show timeline, at 390, or may continue at 320.
[0037] In an exemplary implementation of method 300, the output of the tracking device (e.g., the current and / or relative location of the first tracked UAV and the non-on-board show elements) is used to trigger the operation of non-on-board show effect devices including illumination systems (e.g., spotlights) and / or projectors to illuminate or project onto (e.g., transmit light towards) the UAV or one or more of the UAV's show effect elements (e.g., a projection screen, a surface of the UAV body / landing gear, a covering used to cover the UAV body / landing gear, etc.). This may be considered to achieve a cumulative show effect. In some useful applications, the non-on-board show effect devices operate for a period of time to continue illuminating or projecting onto the UAV or the UAV's show element devices as the relative location of the UAV to the non-on-board show effect elements (tracked in real time by the tracking device) changes (e.g., by operation of the UAV's propulsion mechanism). This may include moving ground spotlights or projections to follow the movement of the UAV in the show airspace. In other embodiments, an illumination system or projector is provided on the first tracked UAV and the projector operates to aim or point at a ground-based or UAV-based non-airborne show effect element (e.g., a projector on the first tracked UAV rotates / pivots to project an image onto a projection screen or surface near the UAV or to a ground location such as on a stage where human or robotic actors perform).
[0038] As noted in step 370, the real-time location of the cooperating UAV and / or its show effect device may be used to automatically adjust the tracking lighting (e.g., lighting provided by a ground-based show effect device or by a device on board the first tracked UAV) by the show control system. For example, the lighting system or projector may be adjusted to minimize the unintended appearance of another UAV, at least temporarily, when the other UAV briefly passes between the off-board show effect device and the first tracked UAV or its show effect device (or between the light source and the intended target object / surface). Such adjustments may result in reducing the brightness (or changing its color) or stopping the lighting stream until the other UAV is no longer illuminated by the lighting stream.
[0039] In some preferred embodiments, show effects outside the vehicle may be adjusted by a main show control system due to real-time vehicle conditions and performance. FIG. 4 illustrates such a control method 400, which may begin at 410, such as with providing a main show control system having a UAV tracking device and / or UAV communication hardware, which may include the main show control system retrieving and initializing a show plan that defines the show timeline and show effects to be generated at specific times along the show timeline. Method 400 continues at 420 with the main show control system operating to monitor and / or track the status and performance of UAVs participating in the show performance. This may include tracking the speed, direction, orientation, and location of the UAV in the show airspace, as well as other operational parameters, such as the current state of each of the UAV's show effect devices (e.g., a tensioned projection screen or object, and a projectile launch mechanism operable in a ready to launch mode at the projection screen or object orientation, etc.).
[0040] Method 400 continues at 430 with the controller operating to compare the UAV's current location with the next of the UAV's mark / cue locations in the show timeline to determine whether the UAV's point (e.g., current operating state) is at or near this location. If not, monitoring and / or tracking of the UAV continues at 420. If so, the controller operates at 440 to trigger a show effect device associated with providing a show effect when the UAV reaches this particular location in the show airspace. This may include triggering ground-based show effect elements, triggering show effect elements on the UAV, triggering show effect devices on a cooperating UAV, and / or cueing a performance by a human or robotic actor to achieve a desired show effect (which may be a cumulative show effect provided by the UAV's interaction with off-board show effect devices). Method 400 then continues at 450 with determining whether the UAV hits the mark / cue in time. If not, an adjustment of the show timeline or show content may be provided at 460 by the show control system, such as when a show effect was generated / performed at an earlier or later time, or without any UAV condition (too far off course to perform the effect, show effect device inoperable, etc.). Method 400 may then end at 490, such as at the end of the show timeline, or may continue with repeating step 420.
[0041] As can be appreciated from this description of Figure 4, show effects outside the UAV may be adjusted based on the real-time conditions and performance of the UAV. In particular, triggering of the operation of show effect devices may occur based on the actual UAV location and / or speed, and not just on a preprogrammed timeline. This allows the main show control system to account for variability in UAV performance, which may vary from show to show due to factors such as wind in the airspace, and hardware characteristics and variability for the UAV.
[0042] Show scenes (and show effects for those scenes) may be triggered earlier or later than ideally intended (or set in the show timeline) by the main show control system based on the actual UAV location compared to expected programming (e.g., the UAV show flight plan). Musical scores, lighting cues, etc. may be bundled into mini-scenes to be triggered and, when useful, mixed based on the actual UAV location to maintain synchronization throughout the show. The result is more cohesive compared to one long show timeline that starts executing when the UAV begins the UAV's show flight plan / profile, which is done regardless of the UAV's true location, which may be slightly off course due to performance irregularities. This content may be written such that it can be gracefully mixed into subsequent scenes in the show timeline and can be stylistically ended earlier or later in the track.
[0043] 5 is a flow diagram of a show system control method 500 for safety-based control of show effect elements based on tracked positions of UAVs in the show airspace. During operation of an air show or display system, such as the system 100 of FIG. 1, status and / or tracking information of the UAVs can be effectively utilized to improve safety of system operation during show performance. The method 500 may begin at 510, such as with defining a set of safety requirements for the show performance, which may include safety requirements for operation / triggering of one or more show effect devices, such as pyrotechnic devices, laser devices, etc. The safety requirements may also include operational steps for the show system or components of the show system upon the occurrence of certain events, such as upon emergency conditions for the UAVs, intrusion of non-show vehicles into the airspace, etc.
[0044] At 520, the method 500 includes a main show control system device or other component operative to track the location of all UAVs in the show airspace. At 530, the method 500 includes determining the next show effect to be triggered in a show timeline or show plan currently being used to perform a show or display in the show airspace. At 540, the method 500 includes a main show control system operative to determine, based on the tracked positions of the UAVs from 520, whether all UAVs in the airspace are in a position that may be safe to perform the next show effect (e.g., launching projectiles, initializing pyrotechnics, and other show effect device operations). If not, at 544, the show effect may be disabled or skipped in the show timeline, or modified (e.g., modifying projectile activation, modifying the amount or type of pyrotechnics, etc.) to allow for safe performance of the show effect. If all UAVs are in a safe position, the show effect device associated with the next show effect is triggered at 548 and method 500 continues at 520. In this manner, the global show control system can use a tracking system / device to ensure that potentially dangerous show effects, such as pyrotechnics or lasers, are not activated when a UAV is in an unsafe location (such as near an audience member or another UAV).
[0045] Concurrent with steps 530-548, method 500 includes monitoring the UAVs in the show system at 550 to determine whether any of the UAVs are reporting (or are determined to have) an abnormal operating condition that requires an emergency landing to be made during the show performance. If not, method 500 continues at 520. If so, method 500 continues at 556 with operating one or more show effect devices to assist in landing the UAV experiencing the emergency operating condition. For example, the show control system may operate a lighting device (e.g., a spotlight) to its maximum brightness and to illuminate (and in some cases track / follow in real-time) the UAV as it makes an emergency landing to improve safety of the show crew and audience during the landing of the UAV.
[0046] Additionally, concurrently with steps 530-548, method 500 includes monitoring the show airspace at 560 for any non-show vehicles. If not detected, method 500 continues at 520. If an unexpected UAV or other substrate is detected at 560, method 500 includes operating one or more show effect devices at 564 as appropriate for the detected intrusion. Method 500 may end at 590, such as at the end of the show timeline. Step 564 may include operating non-aircraft illumination (e.g., spotlights) to illuminate the unexpected or intruded vehicle and track its real-time movement through the airspace over time, and / or to illuminate show UAVs and show effect devices of the show UAVs that may be in the path of the intruded vehicle to reduce the risk of collision with show system components.
[0047] While the present invention has been described and illustrated with a certain degree of particularity, it will be understood that the disclosure is made by way of example only, and that many variations in the combination of parts and configurations may be utilized by those skilled in the art without departing from the spirit and scope of the invention as hereinafter claimed.
[0048] For example, applications exist where the UAV takes the form of an aerial robotic device that may not have its own propulsion mechanism but may instead be dropped from a height or launched into the show airspace. In the progression of the aerial robotic device through the show airspace, it may be useful to track the performance of such aerial robotic device with respect to the timing of the aerial robotic device's progression, the performance of a particular function of the aerial robotic device (e.g., that an on-board show effect device, such as a robotic mechanism, performed as expected at a particular time in the show timeline), and / or the position, orientation, etc. of the aerial robotic device, and this tracked information can be used to trigger on-board and / or off-board show effect devices (synchronously, as described above) to achieve a desired show effect (standalone or cumulative effect).
[0049] Untethered aerial robotic devices are subject to greater performance variability than conventionally powered animatronics. Thus, it is difficult to rely on mere synchronization of pre-programmed behavior initialization to ensure that ground-based effects such as sounds, ground pyrotechnics, etc. (or other non-airborne show effects) occur at the correct time and with the correct settings / tuning based on the performance of the actual aerial robotic device.
[0050] For example, if wind delays the aerial robotic device by even a fraction of a second, a ground-based pyrotechnic show effect device may be fired the moment before the aerial robotic device reaches the appropriate configuration and position in the sky (e.g., a hands-out pose to correspond with a flying superhero, etc.). In another example, the position may be precise at the correct time, but the orientation may differ from the planned show effect ("Show Effect A") such that simply following the original show timeline and planned show effect device operation would be deemed inappropriate (e.g., Show Effect A may be an audio soundtrack playback from a ground-based show effect device, but in the actual orientation and / or pose achieved, it may be desirable to use "Show Effect B," which may be preprogrammed and selected by the show control system, or may be generated in real-time or dynamically in response to the actual UAV's orientation, pose, position, timing, or other parameters). Another example may be ground-based projection or illumination effects necessary to get a flying animatronic figure to be precise, even though the exact position at any given time during the show may change. The inventors have recognized that what is needed is a way to trigger ground-based effects from an untethered aerial robotic device based on the actual forward movement or pose of the aerial robotic device and / or from a ground-based control system based on tracked real-time movement of the aerial robotic device.
[0051] When the UAV is a ballistically launched object in the form of an aerial robotic device (or, in some cases, when the UAV is a quadcopter (or other copter) with on-board propulsion, drone, etc.), there are several techniques that may be used in a show system to trigger non-airborne effects. As a first example, an untethered aerial robotic device may be instrumented with one or more sensors that measure various state variables, such as a point sensor, an inertial measurement unit (IMU), an altitude sensor, an attitude sensor, etc. For example, a ground-based or non-airborne show effect device may be scheduled in the show plan to operate when the ballistic aerial robotic device is at its highest point and / or when the ballistic aerial robotic device is in a particular attitude; however, the ground-based show control system may generate trigger signals appropriate to the measured or sensed state variables instead of simply playing back the show plan and hoping for synchronization.
[0052] In one particular case, the ballistic aerial robotic device is at the highest point of the ballistic aerial robotic device's thrust or launch event when the accelerometer measures zero g, and the show plan may call for a show effect device in the form of an illumination system (e.g., spotlight) to operate to illuminate the aerial robotic device when it is at its highest point or appears to be hovering. An on-board sensor (e.g., sensor 126 in FIG. 1) may be operated to detect zero g, and an on-board communication device (e.g., on-board radio technology) may be operated by an on-board controller to transmit information to a ground-based show control system (e.g., system 170 in FIG. 1), which may respond by generating a trigger signal such that the appropriate show effect device operates in sync with the actual performance of the aerial robotic device. In some cases, trigger or control signals are also generated to suit the orientation of the flying aerial robotic device, such as to select a precise lighting system to illuminate the front (or other) face of the aerial robotic device based on the attitude or orientation sensed / determined by on-board sensors (e.g., to operate a show assembly in real time to provide show effect B or a cumulative effect due to operation of the aerial robotic device and its on-board show effect device).
[0053] In some embodiments, the on-board communication components of the aerial robotic device may operate, such as with the aerial robotic device's control software module, to generate and transmit trigger or control signals to appropriate off-board specific effect devices to achieve a desired show effect or cumulative show effect. Alternatively or additionally, the aerial robotic device may operate its on-board sensors to detect proximity to an object or another UAV (e.g., another aerial robotic device) and use this information to trigger on-board show effect devices and / or ground-based (or off-board) show effect devices to perform pre-programmed functions or functions specifically selected in response to the particular close proximity detected. For example, an airborne collision may be detected that is processed such that a corresponding sound effect is triggered only when two "colliding" (e.g., a near miss that appears to be a collision to onlookers) UAVs are in close proximity.
[0054] Alternatively or additionally, ground-based tracking and visual monitoring may be used to determine when the aerial robotic device is at a precise point and attitude and may be used to trigger ground events (ground-based effect devices that are pre-programmed or programmed in real-time) and this solution may use very fast tracking technology and / or motion prediction algorithms, techniques. When the attitude and / or orientation is determined, an on-board or ground-based show control system may process this information to generate control triggers / signals for the show effect devices to operate as appropriate for that attitude and / or orientation. For example, an audio module may process the attitude and / or orientation of the aerial robotic device (e.g., which way is it facing when the effect is triggered, a flying robot?) and may select or adjust an effect accordingly to suit the attitude and / or orientation. For example, audio clip playback may be adjusted (e.g., by Doppler shifting, etc.) to be more realistic when the aerial robotic device is not facing the audience located below the show system's airspace (e.g., when the aerial robotic device faces away from the audience on the ground, the sound is transmitted away from the audience).
[0055] In the same or other embodiments, show effects are used to create cumulative effects (combinations of ground-based show effect devices and flying or aerial robotic devices or their launch devices) to fake some of the operation of the show system. For example, a mechanical winch may be used to launch or hurl a robotic device (e.g., to provide an aerial robotic device or stunt robot), and an audio system on the UAV and / or in the ground-based show assembly may have its operating time synchronized with the mechanical winch to generate an output sound or sound clip that masks the audio noise of the mechanical winch.
Claims
1. 1. A system configured to achieve dynamic participation in a distributed airshow, comprising: a ground-based show control system that generates trigger signals for triggering show effects based on a timeline defined for the distributed airshow; an unmanned aerial vehicle (UAV), the UAV comprising an on-board or off-board flight propulsion mechanism operable to move the UAV in three dimensions in a show airspace; one or more show effect devices physically separate from the UAV and operable in response to the trigger signal to generate a first show effect; one or more show effect devices physically on or supported by the UAV and operable in response to the trigger signal to generate a second show effect; a tracking device coupled to the ground-based show control system for tracking a relative position of the UAV and the show effect device physically separate from the UAV, the relative position being used by the ground-based show control system to select a time for issuing the trigger signal; The system, wherein the first show effect and the second show effect interact to generate a cumulative show effect.
2. The system of claim 1, wherein the UAV further comprises an on-board show controller having computational resources to execute scripts to generate show control signals for the one or more show effect devices physically on or supported by the UAV based on the show control signals.
3. 3. The system of claim 2, wherein the script includes mini-scenes including musical scores or lighting cues that are triggered and mixed based on the actual position of the UAV in the show airspace to maintain synchronization of the distributed airshow across the timeline, and the mini-scenes are configured to end earlier or later on the timeline to suit the actual position of the UAV.
4. 2. The system of claim 1, wherein the ground-based show control system generates the trigger signal based on the timing of the UAV traveling to or through a location in the show airspace relative to defined timing and mark positions in the timeline for the distributed airshow.
5. The system of claim 1, wherein the relative location is used by the ground-based show control system to generate the trigger signal that causes the one or more show effect devices physically separate from the UAV to illuminate or project toward the UAV as the UAV moves through the show airspace by means of the propulsion mechanism.
6. The system of claim 1, wherein the tracking device determines real-time positions of additional UAVs in the show airspace, and the ground-based show control system automatically modifies operation of the one or more show effect devices to adjust lighting or projection to control lighting of one or more of the additional UAVs located between the UAV and the one or more show effect devices physically separate from the UAV.
7. The system of claim 4 , wherein the trigger signal is generated earlier or later than the timing defined in the timeline for the distributed airshow.
8. The system of claim 1, wherein the one or more show effect devices physically on or supported by the UAV provide light, video, or laser projection from the UAV to the one or more show effect devices physically separate from the UAV, or the one or more show effect devices physically separate from the UAV provide light, video, or laser projection to or toward the one or more show effect devices physically on or supported by the UAV.
9. The system of claim 1, wherein the one or more show effect devices physically separate from the UAV launch projectiles toward the UAV, or the one or more show effect devices physically on or supported by the UAV launch projectiles toward the one or more show effect devices physically separate from the UAV.
Citation Information
Patent Citations
Self charging lightweight drone apparatus
US10424231B2
Aerial display system with floating projection screens
US20140233099A1
Controlling unmanned aerial vehicles as a flock to synchronize flight in aerial displays
US20140249693A1
Systems and methods for unmanned aerial vehicles
US20190112048A1
Adaptive Voxels for Aerial Light Shows
US20190315486A1
