Advanced unmanned aerial vehicles including voice synchronization and geofencing

Abstract

The capabilities of an unmanned aerial vehicle (UAV) may be location-dependent. In some embodiments, a digital audio communication channel is synchronized and multiplexed with a high-resolution video stream captured from a user-controlled remote UAV. The accompanying audio stream is used to provide supplemental and enriching information to the end user controlling the actions of the remote UAV. This system can be used to provide virtualized tourism experiences at places of significant interest, such as historic settlements, wildlife refuges, and iconic destinations. In some embodiments, a virtual three-dimensional geometry with a bounding surface, commonly referred to as a geofence, and an accompanying control system restrict the flight path of the remotely piloted UAV to the interior geometry.

Description

[Technical Field]

[0001] Background technology Quadcopter UAVs, also known as drones, are often equipped with high-resolution video cameras capable of transmitting real-time captured digital video to a remote controller device. International Patent Application No. PCT / US2021 / 032011 also describes a system and method for transmitting a real-time video stream over a wide area network (WAN) to a remote end user who can control the UAV, including both the drone's movement and the direction and focus of an attached imaging device. See International Patent Application No. PCT / US2021 / 032011, filed May 12, 2021, the entire contents of which are hereby incorporated by reference.

[0002] Quadcopter UAVs are becoming increasingly common for commercial and personal use. As the frequency and volume of drone activity increases, airspace congestion is exponentially increasing, resulting in a variety of adverse effects, including a high risk of mid-air collisions and ground damage to collateral property and people, as well as increased noise disturbance to humans and wildlife. To mitigate these negative effects, national aviation authorities have enacted laws regulating the airspace over which drones may fly. These restrictions are most often concentrated in and around areas with high air traffic, noise-sensitive areas such as national parks, and over public stadiums and other locations where large numbers of people gather. Most drones are now equipped with imaging devices such as video cameras, and aviation authorities are also restricting drone airspace over and around areas of national interest, such as military bases, government buildings, public facilities, nuclear power plants, and dams. Furthermore, legal challenges regarding privacy and copyright infringement are further driving drone airspace restrictions. Summary of the Invention

[0003] This Summary introduces a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope.

[0004] A variety of advanced functions for unmanned aerial vehicles (UAVs) can be provided, such as the ability to be determined by the UAV's position, synchronizing a real-time video stream from the UAV with audio data from an additional source, or remotely taking over control of the UAV's movements by another individual or system.

[0005] In some embodiments, a digital audio communication channel is captured by a user-controlled remote unmanned aircraft vehicle and synchronized and multiplexed with the video stream transmitted by the UAV to provide a virtualized travel experience. The audio data provided by the digital audio communication channel can be provided by capturing a live narration from an individual, by a computing device that creates the audio data, or by a computing device that selects from recorded audio data based on information about the location of the UAV or objects in the real-time video stream from the UAV. The accompanying audio stream is used to provide supplemental and enriching information to an end user controlling the actions of the remote UAV. This system can be used to provide a virtualized tourism experience at locations of significant interest, such as historic settlements, wildlife refuges, and iconic destinations. The person narrating the real-time video stream from the UAV can take over control of the UAV from the end user.

[0006] In some embodiments, a three-dimensional spatial region defines an airspace permitted for operating an unmanned aerial vehicle. The position of the UAV may be input to a control system for the UAV and may be used to maintain the UAV within the airspace permitted. In some embodiments, a virtual three-dimensional geometry with a bounding surface, commonly referred to as a geofence, and an associated control system restricts the flight path of a remotely piloted unmanned aerial vehicle ("UAV") to a geometry within the geofence. The control system may take over control of the UAV from an end user to ensure that the UAV remains within the geometry within the geofence.

[0007] Thus, in one aspect, a computing device is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to the wide area network and receives instructions from an end user computing device to generate control data for the unmanned aerial vehicle. The remote controller device transmits a video feed based on the real-time imaging data over the wide area network. A computing device for use in such a system includes a processing system including processing circuitry and a memory that stores computer program instructions. The computer program instructions configure the computing device to track spatial coordinates related to the geographic location of the unmanned aerial vehicle and generate geographic location-based information of the location of the unmanned aerial vehicle for transmission to the end user computing device.

[0008] In one aspect, a method is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to the wide area network and receives instructions from an end user computing device to generate control data for the unmanned aerial vehicle. The remote controller device transmits the real-time imaging data as a video transmission over the wide area network. A method for use in such a system includes tracking spatial coordinates related to the geographic location of the unmanned aerial vehicle and generating geographic location-based information of the location of the unmanned aerial vehicle for transmission to the end user computing device.

[0009] In one aspect, a system includes an unmanned aerial vehicle, a remote controller device, an end-user computing device, and an audio-guided computing device. The unmanned aerial vehicle and the remote controller communicate via a wireless network. The remote controller device, the end-user computing device, and the audio-guided computing device communicate via a wide-area network. The unmanned aerial vehicle includes an imaging sensor and a radio. The radio is connected to transmit real-time imaging data from the imaging sensor over the wireless network and to receive control data from the wireless network. The remote controller device includes a radio tuned to receive real-time imaging data from the unmanned aerial vehicle over the wireless network. The remote controller device is configured to transmit control data to the unmanned aerial vehicle over the wireless network and transmit video transmissions based on the real-time image data received over the wide-area network. The end-user computing device includes one or more presentation devices. The end-user computing device is configured to transmit instructions to the remote controller device to prepare control data for transmission to the unmanned aerial vehicle. The audio-guided computing device transmits supplemental audio data. The end-user computing devices receive the synchronized and multiplexed video transmission and supplemental audio data over a communication channel of the wide area network and present the received video transmission and supplemental audio data via one or more presentation devices.

[0010] In one aspect, a method includes transmitting, by the unmanned aerial vehicle, real-time imaging data from an imaging sensor of the unmanned aerial vehicle over a wireless network; receiving, by the unmanned aerial vehicle, control data from the wireless network by the unmanned aerial vehicle over the wireless network; receiving, by a remote controller device, the real-time imaging data from the wireless network; transmitting, by the remote controller device over the wireless network, instructions to the remote controller device to create control data for the unmanned aerial vehicle over the wide area network by an end user computing device; preparing, by an audio guided computing device, supplemental audio data; synchronizing and multiplexing, over a communication channel of the wide area network, a video transmission and supplemental audio data based on the real-time imaging data for transmission to the end user computing device over the communication channel; and presenting the video transmission and supplemental audio data via one or more presentation devices of the end user computing device.

[0011] In one aspect, an audio-guided computing device is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to the wide area network and receives instructions from an end user computing device to generate control data for the unmanned aerial vehicle. The remote controller device transmits a video feed based on the real-time imaging data over the wide area network. In such a system, the audio-guided computing device includes a processing system including processing circuitry and a memory that stores computer program instructions. The computer program instructions configure the audio-guided computing device to provide supplemental audio data and to synchronize and multiplex the video feed and the supplemental audio data over a communication channel of the wide area network for transmission to the end user computing device.

[0012] In one aspect, a method is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to the wide area network and receives instructions from an end user computing device to generate control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide area network. A method for use in such a system includes providing supplemental audio data and synchronizing and multiplexing the video transmission and the supplemental audio data over a communication channel of the wide area network for transmission to the end user computing device.

[0013] In one aspect, a system includes an unmanned aerial vehicle, a remote controller device, an end user computing device, and a geofencing computing device. The unmanned aerial vehicle and the remote controller device communicate over a wireless network. The remote controller device, the end user computing device, and the geofencing computing device communicate over a wide area network. The unmanned aerial vehicle includes an imaging sensor and a radio. The radio is connected to transmit real-time imaging data from the imaging sensor over the wireless network and to receive control data from the wireless network. The remote controller device includes a radio tuned to receive the real-time imaging data from the unmanned aerial vehicle over the wireless network. The remote controller device transmits control data to the unmanned aerial vehicle over the wireless network and transmits received video feeds based on the real-time image data over the wide area network. The end user computing device includes one or more presentation devices. The end user computing device is configured to transmit instructions to the remote controller device to prepare control data for transmission to the unmanned aerial vehicle. The geofencing computing device creates a representation of a three-dimensional spatial region based on a collection of polygonal prisms for defining an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence, and tracks spatial coordinates related to the geographic location of the unmanned aerial vehicle. The geofencing computing device determines a spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

[0014] In one aspect, a method includes transmitting, by the unmanned aerial vehicle, real-time imaging data from an imaging sensor of the unmanned aerial vehicle over a wireless network; receiving, by the unmanned aerial vehicle over the wireless network, control data from the wireless network; receiving, by a remote controller device, the real-time imaging data from the wireless network; and transmitting, by the remote controller device over the wireless network, the control data; transmitting, by an end-user computing device over the wide area network, instructions to the remote controller device to create the control data for the unmanned aerial vehicle; creating a representation of a three-dimensional spatial region based on a collection of polygonal prisms to define an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence; tracking spatial coordinates related to the geographic location of the unmanned aerial vehicle; and determining a spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

[0015] In one aspect, a geofencing computing device is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from a remote controller device from the wireless network. The remote controller device is connected to the wide area network and receives instructions from an end user computing device to generate control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide area network. The geofencing computing device includes a processing system including a processing circuit and a memory that stores computer program instructions. The computer program instructions configure the geofencing computing device to create a representation of a three-dimensional spatial region based on a collection of polygonal prisms that define an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence. The geofencing computing device further tracks spatial coordinates related to the geographic location of the unmanned aerial vehicle. The geofencing computing device further determines a spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

[0016] In one aspect, a method is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to the wide area network and receives instructions from an end user computing device to create control data for the unmanned aerial vehicle. The remote controller device transmits a video feed based on the real-time imaging data over the wide area network. The method includes creating a representation of a three-dimensional region of space based on a collection of polygonal prisms to define an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence. The method includes tracking spatial coordinates related to the geographic location of the unmanned aerial vehicle. The method includes determining a spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

[0017] In one aspect, a geofencing computing device is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from a remote controller device from the wireless network. The remote controller device is connected to a wide area network and receives instructions from an end user computing device to create control data for the unmanned aerial vehicle. The remote controller device transmits the real-time imaging data as a video transmission over the wide area network. The geofencing computing device includes a processing system including a processing circuit and a memory that stores computer program instructions. The computer program instructions configure the geofencing computing device to create a representation of a three-dimensional spatial region that defines an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence. The minimum above ground level of each point inside the geofence is determined by a visibility analysis based on the three-dimensional coordinates of the geographic location of the remote controller device and an elevation map of the geographical terrain of the geofence area. The geofencing computing device tracks the three-dimensional spatial coordinates of the geographic location of the unmanned aerial vehicle. The geofencing computing device determines a spatial relationship between the geographic location of the unmanned aerial vehicle's position and the geofence.

[0018] In one aspect, a method is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to a wide area network and receives instructions from an end user computing device to create control data for the unmanned aerial vehicle. The remote controller device transmits the real-time imaging data as a video transmission over the wide area network. The method includes creating a representation of a three-dimensional spatial region defining an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence. The minimum above ground level of each point inside the geofence is determined by a visibility analysis based on 1) the three-dimensional coordinates of the geographic location of the remote controller device and 2) an elevation map of the geographic terrain of the geofenced area. The method includes tracking the three-dimensional spatial coordinates of the geographic location of the unmanned aerial vehicle. The method includes determining a spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

[0019] In one aspect, a computing device is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to a wide area network and receives instructions from an end user computing device to generate control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide area network. The computing device includes a processing system including processing circuitry and a memory that stores computer program instructions. The computer program instructions configure the computing device to generate text describing the content of the video transmission based on a large-scale language model. The computing device generates audio data corresponding to the generated text using a speech synthesis engine. The supplemental audio data is synchronized and multiplexed with the video transmission over a communication channel of the wide area network for transmission to the end user computing device.

[0020] In one aspect, a method is for use in a system involving a remotely controlled unmanned aerial vehicle. The unmanned aerial vehicle includes an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device. The remote controller device is connected to the wide area network and receives instructions from an end user computing device to generate control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide area network. The method includes generating text describing the content of the video transmission using a large-scale language model. The method includes generating audio data corresponding to the generated text using a speech synthesis engine. The method includes synchronizing and multiplexing the video transmission and the audio data over a communication channel of the wide area network for transmission to the end user computing device.

[0021] All of the above aspects may include one or more of the following features: the unmanned aerial vehicle is a quadcopter drone; a low-earth orbit satellite provides a direct wireless link from the remote controller device to the unmanned aerial vehicle; a mobile communications tower provides a direct wireless link between the remote controller device and the unmanned aerial vehicle; and the remote controller device is configured to act as a network bridge between a wireless network and a wide area network.

[0022] All of the above aspects may include one or more of the following characteristics: One or more additional audio guide computing devices are connected to a wide area network, each providing respective supplemental audio data synchronized and multiplexed with the video transmission from the unmanned aerial vehicle; The supplemental audio data includes live audio data of an audio guide narration captured via a microphone for the audio guide's audio guide computing device; The respective supplemental audio data transmitted by the one or more additional audio guide computing devices includes data from a local file on the additional audio guide computing device; The transmission of the supplemental audio data is effected by the geographic location of the unmanned aerial vehicle; The remote controller device incorporates the audio guide computing device.

[0023] All of the above aspects may include one or more of the following features: The artificial intelligence system is configured to identify an image signature of an object at the location of the unmanned aerial vehicle using real-time imaging data; The artificial intelligence system is configured to control the unmanned aerial vehicle to track the object having the identified image signature; The artificial intelligence system is configured to control the unmanned aerial vehicle and the imaging sensor to focus on the object having the identified image signature; The artificial intelligence system is configured to insert a graphic overlay onto the video transmission based on the object having the identified image signature; The artificial intelligence system is configured to cause the transmission of audio information related to the object having the identified image signature.

[0024] All of the above aspects may include one or more of the following characteristics: The artificial intelligence system or audio guidance computing device may include a large-scale language model configured to generate text describing the content of the video transmission, the speech synthesis engine generates speech corresponding to the generated text, and the supplemental audio data includes the generated speech.

[0025] All of the above aspects may include one or more of the following characteristics: A notification is generated to the end user computing device based on the determined spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

[0026] In any of the above, the minimum value above the ground surface for each point inside the geofence may be determined by a visibility analysis based on 1) three-dimensional coordinates of the geographic location of the remote controller device and 2) an elevation map of the geographic terrain of the geofence area.

[0027] All of the above aspects may include one or more of the following characteristics: Determining the spatial relationship includes determining whether the geographic location of the unmanned aerial vehicle is in exterior space or near a geofence; Creating the notification includes storing a minimum threshold time and a minimum threshold distance, and triggering the notification based on either (i) the distance of the unmanned aerial vehicle to the geofence being less than or equal to the minimum threshold distance, or (ii) the predicted time for the unmanned aerial vehicle to reach the geofence being less than or equal to the minimum threshold time; The minimum threshold time and minimum threshold distance for triggering the notification are computed values ​​based on the kinematics of the unmanned aerial vehicle; The minimum threshold distance is based on a maximum deceleration of the unmanned aerial vehicle in a direction perpendicular to the geofence; and The minimum threshold time is based on a maximum deceleration of the unmanned aerial vehicle in a direction perpendicular to the geofence.

[0028] All of the above aspects may include one or more of the following characteristics: The notification includes a graphical representation superimposed on a video transmission sent from the remote controller device to the end user computing device.

[0029] All of the above aspects may include one or more of the following characteristics: The remote controller device is caused to send control data to the unmanned aerial vehicle to decelerate away from the geofence to avoid navigating into external space, where the remote controller is caused to do so after invoking a notification.

[0030] All of the above aspects may be embodied as a computer system, as any individual component of the computer system, as a process performed by the computer system or any individual component of the computer system, or as an article of manufacture including computer storage on which computer program code is stored and which, when processed by one or more computer processing systems, provides the computer system or individual component of the computer system or configures one or more computer processing systems to perform the method.

[0031] The following detailed description refers to the accompanying drawings which form a part of this application and which show, by way of illustration, implementations of specific embodiments. Other implementations may be made without departing from the scope of the present disclosure.

[0032] Reference is now made to the accompanying drawings, which are not necessarily drawn to scale, and some embodiments may include fewer or more components than shown in the drawings. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 illustrates a system for audio and video narrated virtual tours of remote destinations navigated by end-user controlled drones. [Figure 2] 1 illustrates a block diagram of an exemplary circuit embodying a computing device that may perform various operations in accordance with some example embodiments described herein. [Figure 3] FIG. 1 illustrates a system for audio and video narrated virtual tours of remote destinations navigated by end-user controlled drones. [Figure 4] Shows the geometric interpretation of the voxels or rectangular prisms used to create the interior region. [Figure 5]10 depicts the geometry of multiple voxels used to define an example of an allowable airspace for UAV navigation over terrain. [Figure 6] , shows an exemplary diagram for superimposition onto a UAV when reaching the voxel side of the boundary. [Figure 7] 1 illustrates a color-coded terrain map showing an exemplary geofence. [Figure 8] 10 is a graph of an example output result of the addViewshedPath function, where the orange line is the minimum AGL created in this radial direction. DETAILED DESCRIPTION OF THE INVENTION

[0034] A variety of advanced functions for unmanned aerial vehicles (UAVs) can be provided, such as the ability to be determined by the UAV's position, synchronizing a real-time video stream from the UAV with audio data from an additional source, or remotely taking over control of the UAV's movements by another individual or system.

[0035] First, let's look at audio synchronization.

[0036] Quadcopter UAVs, also known as drones, may be equipped with a video camera capable of transmitting real-time captured digital video to a remote controller device. As described in International Patent Application No. PCT / US2021 / 032011, which is incorporated herein by reference, the real-time video stream can be transmitted from the UAV to a remote end user over a wide area network (WAN). The video camera can be a high-definition video camera. The remote end user can operate the UAV controls, including both drone movement and the direction and focus of the attached imaging device. A digital audio communication channel is synchronized and multiplexed with the video stream captured from the user-controlled remote unmanned aircraft vehicle. The accompanying audio stream is used to provide supplemental and enriching information to the end user controlling the operation of the remote UAV. This system can be used to provide virtual tourist experiences at locations of significant interest, such as historic settlements, wildlife refuges, and iconic destinations.

[0037] Using an accompanying audio guide that provides real-time commentary, end users can be guided to specific areas of interest, presented with supplemental information about the drone's location, and engaged in informative dialogue with one or more audio guide experts who access the UAV's video feed. Additionally, the accompanying audio guide can temporarily control the UAV's movement and its imaging device to efficiently direct the end user's attention to the area of ​​interest. In another implementation, a stored audio file can be transmitted when the UAV is flying near the area of ​​interest, or an AI image recognition system can cause the audio transmission after identifying a relevant image signature in the UAV's transmitted video. The present invention is not limited to audio data, as other media data can be created or provided based on the UAV's location.

[0038] Figure 1 depicts an exemplary system for audio- and video-narrated virtual tours of remote destinations navigated by an end-user-controlled drone. 100 is an unmanned aerial vehicle (UAV) preferably equipped with a high-resolution video camera and means for transmitting captured video data over a wireless network.

[0039] 101 is a remote controller device that communicates with the UAV (100) and acts as a network bridge between the UAV (100) and a wide area network (102). 102 is the wide area network commonly referred to as the Internet. Management system 103 is a component within the network that manages, provides, and stores critical data and authentication information used by the operation of the system. Management system 103 may contain data for identifying digital audio and visual signatures of data transmitted by the unmanned aerial vehicle and may exchange that data with AI system 106, such as those described below.

[0040] 104 is an end-user computing device, such as a PC or mobile device, connected to the wide area network 102, that runs a software application that transmits user-control information to the UAV 100. The end-user computing device 104 accesses the voice communication channel and receives video streaming from the UAV 100, all through a proxy provided by the remote controller device 101. There may be multiple end-user computing devices.

[0041] 105 is an audio guided computing device with a network connection to wide area network 102. This computing device runs software applications that transmit supplemental audio data to end user computing devices 104, management system 103, and remote controller device 101. The audio guided computing device also receives data from management system 103 over WAN 102. In some embodiments, multiple interconnected audio guided computing devices 105 may be used.

[0042] One class of unmanned aerial vehicle (UAV) is the quadcopter. Drones have become commonplace for use by both businesses and consumers. Most drones are equipped with digital imaging devices that produce high-resolution digital images and videos. These digital images can be stored on a storage device onboard the drone and, in most implementations, transmitted back to a remote controller device over a wireless network link. Commercial entities may use these drones and their imaging capabilities in dangerous or physically challenging environments, such as investigating ongoing forest fires, checking for gas pipeline leaks, or actively monitoring and tracking wildlife poachers.

[0043] In some implementations, the remote controller device acts as a network bridge between the drone's attached wireless network and a wide area network (WAN). In this case, the drone's transmitted real-time image data is proxied to a remote end-user device over the WAN by the remote controller. The end-user can view this real-time video data from the remote device and transmit control information, such as changes to flight path or camera direction and focus, back to the bridged drone by the remote device remote controller.

[0044] Embodiments involving audio guidance enhance the drone-transmitted video experience with the addition of an audio communication channel over the WAN accessible by remote controller devices, end user devices, and one or more other computing devices primarily used to transmit supplemental audio data synchronized and multiplexed with the drone-transmitted video stream.

[0045] This configuration enables virtual sightseeing scenarios for end users when the computing device is used with an audio guide that can assist and inform the end user of various high-value points of interest that coincide with the drone's flight path. In this configuration, the end user may also communicate with a person operating a remote control unit who may 1) participate as the audio guide of the flight, 2) act as a remote pilot who can regain control of the drone in case of an emergency, and 3) act as a visual spotter who communicates local information such as the proximity of other aircraft, a flock of birds entering the drone's airspace, or a weather front moving into the area.

[0046] In this same configuration, in some implementations, the device used by the audio guide is provided priority access to the drone controls, allowing the audio guide to remotely control the drone's movement and the direction and lens magnification of its imaging device, such as a camera. A use case for this configuration is an audio guide who is identifying fast-moving animals or locating highly camouflaged animals in dense flora and wants to guide the video experience to these areas of interest.

[0047] In some implementations, the audio guidance communication is stored on the audio guidance device and is replaced by contextually relevant pre-stored audio files, each of which individually corresponds to information related to a particular point of interest within a geofenced boundary of the drone's flight path. When the drone's flight path is geographically proximate to the point of interest, the audio guidance device streams the geographically appropriate pre-recorded audio to the end user device over the WAN.

[0048] These points of interest are then determined during flight by an AI-based image processing system. The AI-based image processing system uses an object detection model that is pre-trained on defined object classes, such as various animals. The system processes each captured video frame to map image signatures to object classes with a confidence factor. If the confidence factor exceeds a predetermined threshold, the system generates a notification that the image signature matches a particular object class. If the image signature matches multiple object classes, the system selects the object class with the highest confidence level to identify the object. An example of this algorithm, written in Python using OpenCV (open-source computer vision) and machine learning software libraries, and the YOLO library for object detection, is as follows:

number

[0049] After the AI ​​image processing system identifies image signatures in the drone's streaming video transmission as classes of interest, the AI ​​system can 1) navigate the drone to the identified area of ​​interest, 2) control the drone's camera to focus on the identified area of ​​interest, 3) embed a graphic overlay in the video transmission to assist in identifying the area of ​​interest, and 4) provide the end user with the transmission of a contextually relevant pre-recorded audio stream.

[0050] For example, a group of sleeping lions camouflaged by dense undergrowth may be identified in a drone video feed by the AI ​​system, which navigates the drone to hover over the sleeping lions, focus the video camera on the group, highlight the group in a diagram in the video feed to assist the end user in identification, and transmit an associated pre-recorded audio stream describing the lions' sleep cycles.

[0051] In some configurations, the human audio description narration provided by the audio description computing device is replaced by a large-scale language model (“LLM”) combined with a speech synthesis engine. In some implementations, the LLM-based narration is provided by the AI ​​system described above used to identify image signatures in drone-streamed video feeds. The generation of the LLM narration and text-to-speech conversion may be performed on a computing device supporting the AI ​​system 106, a computing device supporting the audio description computing device 105, or another computing device.

[0052] To enable this functionality to provide narration of a large language model (LLM) based on an AI-based object classification model, the following Python code illustrates an implementation that uses the PYTTSx3 speech synthesis library, OpenAI's LLM, the OpenCV machine learning library, and the YOLO library for object detection.

number

[0053] In all of the above configurations, the end user receives an audio-narrated video of the virtual sightseeing experience.

[0054] Where audio and video streams may be created by multiple sources, any one of the devices mentioned above (remote control unit, AI system, audio-guided computing device, LLM-created narration and speech synthesis system, and end-user device) or a dedicated device in a WAN-connected public or private cloud infrastructure component, such as the central system management server (103) shown in Figure 1, may be used to multiplex the various streams into a single file for local storage. The archived file digitally preserves the end-user's audio and video experience, which may be downloaded over the WAN to the end-user's device for offline playback and / or downloaded (shared) to other devices authorized by the end-user.

[0055] Drone service providers offer drones for remotely piloted flights over scenic destinations, wildlife refuges, historical sites, etc. To provide complementary audio accompaniment synchronized with the video transmission from the UAV, the following components are used:

[0056] 1. A drone equipped with an imaging sensor, such as a high-resolution video camera, and a radio for transmitting imaging data captured by the drone and receiving control data for guiding navigation information for the imaging sensor and the drone.

[0057] 2. Remote Controller Device: For remote controller devices that communicate with drones using radios operating in unlicensed frequency bands (ISM), such as the 2.4 GHz and 5.8 GHz frequencies, most countries have enacted regulations limiting the effective effective radiated power (EIRP) of the radios to minimize interference with other radios operating in the same ISM bands. The EIRP limitation means that the effective spatial range for radio transmission and reception from the remote controller to the drone is limited; noise on the signal saturates the communication link, rendering communication and control ineffective. To extend the spatial range for communication, the remote controller can be connected to a wide area network (WAN) and can support a network that bridges the WAN and wireless network functionality. If the drone's radio uses a different frequency band with less stringent EIRP restrictions, the drone may be directly connected to the WAN, for example, via a low-earth-orbit satellite or a remote mobile communications tower.

[0058] 3. End User Computing Device: An end user may use a computing device, such as a PC or mobile device, connected to the WAN to run a software application that transmits control information to the drone, connects to the voice channel, and receives video streaming data from the drone, all with all network data traffic bridged by the remote controller device described above.

[0059] 4. Remote Audio Guided Computing Device: One or more computing devices connected to a WAN and capable of accessing the drone's video feed via an application, remotely controlling the drone's movements and camera, and having access to communication channels connected to end user computing devices and remote controllers.

[0060] Here we present geofencing for UAVs.

[0061] The allowed drone airspace or interior geometry may be defined by a three-dimensional space bounded by a virtual two-dimensional boundary called a virtual geofence. In embodiments involving a three-dimensional geometry that generally coincides with the geometry of the defined controlled airspace and other simply connected convex boundaries, the interior geometry may be roughly constructed by a collection of polygons. A particular implementation of these polygonal prisms is a rectangular prism commonly referred to as a voxel. The simple geometry of the voxel is useful in determining which polygonal prism contains the drone. A drone is considered to be within the allowed airspace if its location is within one of the polygonal prisms; similarly, a drone is considered to be in the restricted exterior region if it is not located within one of the polygonal prisms.

[0062] In embodiments incorporating geofencing, the system and method 1) creates a virtual geofence airspace defined as the boundary of a collection of polygonal prisms, 2) assumes UAV flight control to ensure containment within the interior geometry, i.e., the permitted airspace, and 3) provides graphical notification of the UAV's proximity to the exterior geometry, i.e., the restricted airspace.

[0063] As UAV airspace density increases, aviation authorities impose drone airspace restrictions to 1) mitigate collisions with other aircraft and damage to people, wildlife, and property; 2) reduce noise pollution in sensitive areas such as eagle nests where chicks are being raised; and 3) avoid flying over or near areas of national interest such as military bases.

[0064] Other reasons for imposing drone airspace restrictions include avoiding collisions with physical obstacles such as trees, mountains, and pylons; reducing the invasion of privacy caused by UAV image capture; and assisting UAV remote pilots in navigating around areas of interest such as iconic waterfalls or wildlife watering holes.

[0065] FIG. 3 illustrates an exemplary system for audio and video narrated virtual tours of remote destinations navigated by an end-user controlled drone 300, preferably an unmanned aerial vehicle equipped with a high-resolution video camera, and a means for transmitting the captured video data over a wireless network.

[0066] 301 is a remote controller device that communicates with the UAV (300) and acts as a network bridge between the UAV (300) and a wide area network (302). 302 is the wide area network commonly referred to as the Internet. Management system 303 is a component within the network that manages, provides, and stores important data and authentication information used by the operation of the system. This management system may contain or exchange data with a geofencing system 306, such as the one described below. The geofencing system is used to: 1) store a representation of exterior, interior, and boundary areas for UAV (300) navigation; 2) track the spatial coordinates of the unmanned aerial vehicle; 3) trigger notifications when the UAV's location is in exterior space or near a geofence; 4) generate graphical warnings overlaid on the UAV's transmitted video; and 5) send control information to the UAV to avoid navigating through a geofence.

[0067] 304 is an end-user computing device, such as a PC or mobile device connected to the wide area network 302, running a software application that transmits user control information to the UAV 300. The end-user computing device accesses the voice communication channel and receives video streaming from the UAV 300, all through the remote controller device 301, which acts as a proxy. There may be multiple end-user computing devices.

[0068] 305 is a voice-guided computing device with a network connection to wide area network 302. This computing device runs software applications that transmit supplemental audio data to end user computing devices 304, management system 303, and remote controller device 301. The voice-guided computing device also receives data from management system 303 over WAN 302. In some embodiments, multiple interconnected voice-guided computing devices may be used.

[0069] Figure 4 shows the geometric interpretation of a voxel or rectangular prism used to create the interior region. 400 represents a voxel with a top voxel base (401) and a bottom voxel base (402). 402 is a 2-tuple, i.e., a point represented by latitude and longitude coordinates, which is the projection of the voxel's vertex onto two faces.

[0070] Figure 5 illustrates the geometry of multiple voxels used to define an example of allowable airspace for UAV navigation over terrain. 500 is the voxel side of the boundary, the side with no other adjacent voxels, and represents the area of ​​the geofence.

[0071] FIG. 6 shows an example diagram (600) for overlay on the UAV (300) as it reaches the voxel side of the boundary.

[0072] The method for constructing a set of voxels to define the interior geometry is to 1) create a heading sampling map that projects the interior geometry of a 3D convex surface onto the Earth's latitude and longitude. This projection map creates a set of 2-tuples, i.e., {(latl,longl), (lat2,long2), etc.}, where each 2-tuple can relate to the minimum and maximum altitude above ground level (AGL) of a virtual geofence. A voxel is created by associating the 2-tuple with a minimum AGL, defined as the average of the minimum AGLs of a selected point and its three nearest neighbors, as the base of the voxel. Similarly, the base of the top of the voxel is created in a similar way, except that it is calculated by averaging the maximum AGLs of the four points. In a spherical coordinate system, the vertices of a voxel are defined as follows:

number

[0073] To determine whether a drone is located within the permitted airspace, the system checks whether the drone's spatial coordinates are contained within one of the defined voxels. If the drone's coordinates are not located within any of the voxels, the drone is considered to be flying outside the permitted airspace. To define a virtual geofence, the concept of a boundary voxel side is introduced. A boundary voxel side is a side that is not adjacent to another voxel. For example, in the current implementation, the base of the top and bottom of all voxels are boundary voxel sides.

[0074] Using the drone's current speed and direction, a method can be used to predict when the drone will be circling within the external geometry, i.e., restricted airspace, which means that the drone is projecting to cross the voxel side of the boundary. Let V(A) be the velocity vector, and A = (LAT a ,LONG a ,ALT a) reaches the voxel side of the boundary defined by the four vertices:

number

[0075] A method is used to calculate the distance and predicted elapsed time for the drone to reach the geofence. The first step is a basic exercise to calculate the distance of the drone, point A, to the voxel side of the boundary defined by P1, P2, P3, and P4 above. 1. Two vectors can be formed from these four coplanar points; e.g.

number

number

number

number

number

number

[0076] The predicted time to reach the virtual geofence can be calculated by dividing the normal component of the velocity vector (the component of the velocity vector perpendicular to the voxel side of the boundary) by the calculated normal distance. Because the drone may have positive velocity components that could potentially intercept up to three additional voxel sides of the boundary, the predicted time calculation needs to be performed on those voxel sides of the boundary where the drone has positive velocity components in the direction of the boundary normal vector. The predicted time to reach the virtual geofence is the minimum of the predicted times associated with reaching any of the individual voxel sides of the boundary.

[0077] A threshold time for the boundary may be defined and used as a benchmark for triggering an alert notification. For example, an alert may be generated if the predicted time to reach the virtual geofence is less than the defined threshold time. In another implementation, the threshold time for the boundary may be calculated simultaneously based on the drone's speed perpendicular to the voxel side of the boundary, the drone's maximum deceleration rate, and the pilot's response time.

[0078] To avoid breaching the geofence, the drone pilot must change the drone's velocity perpendicular to the voxel side of the boundary to either zero (no further progress towards the boundary) or negative (change in direction away from the boundary). For example, if the drone's normal distance to the voxel side of the boundary is D drone and the velocity perpendicular to the voxel side of the boundary is V drone and the maximum drone deceleration is Deacc drone and the notification threshold time is T threshold and the time it takes to reach the virtual geofence is T geofence teeth,

number

[0079] In some embodiments, the notification technique determines the threshold time, T threshold is based on a threshold distance defined by converting

number

[0080] When a notification is issued regarding a pending virtual geofence violation, the alert may take the form of a large warning signal transmitted to the pilot's remote controller device and / or a graphical image of a visual net or "cage" diagram superimposed on the video displayed on the pilot's remote controller device.

[0081] To eliminate variability in pilot reaction times to correct a pending incursion into restricted airspace, the system may assume control through navigation of the drone to redirect it away from the geofence or slow the drone's speed so that it gracefully stops at the geofence and awaits control by a remote drone pilot who will safely navigate away from the geofence.

[0082] The upper altitude limit for drones is often regulated by government aviation authorities as a certain maximum above ground level (AGL). In practice, the AGL of the upper voxel base is usually set to have this maximum altitude. The AGL of the lower voxel base is determined by a visibility analysis between the drone and the remote control unit. To maintain continuous communication, the wireless transmission link between the drone and the remote control unit must be based on a Fresnel zone, free from obstacles such as physical obstructions due to terrain changes or destructive signal interference due to reflections from the terrain.

[0083] The AGL of the base of the lower voxel is determined by a visibility analysis based on 1) the location of the remote controller unit (x0,y0,h) and 2) a mapping of the terrain of the entire geofenced area providing an elevation map H(x,y). The output of this analysis is the minimum AGL defined at all points within the interior of the geofence.

[0084] In the code below, frm=(x0,y0), the position of the remote controller unit; to=(x,y) is the boundary of the geofence at a particular angular direction; dem is an array containing the terrain elevation H(x,y) between frm and to; CLEARANCE is the minimum offset above the ground to avoid obstacles such as trees, buildings, etc.; RADIO_ELEV is h, the altitude of the remote control unit.

number

[0085] The output_array is a sequence of points in the radial direction between the remote controller and the sampled geofence boundary. As an example, in FIG. 7, the endpoints of the red line 702 imposed on the terrain map 700 correspond to the location of the remote controller and the edge of the geofence boundary. FIG. 8 is a graph 800 of the output result of the addViewshedPath function, where the orange line 802 is the minimum AGL created in this radial direction. To create a complete sub-AGL map of the interior of the geofence, ddViewshedPath is called for further samples of the geofence boundary.

[0086] Exemplary Management System

[0087] The various components of Figures 1 and 3, namely, management system 103, remote controller device 101, UAV 100, AI system 106, voice-guided computing device, end-user computing device 104, management system 303, remote controller device 301, UAV 300, geofencing system 306, voice-guided computing device 305, and end-user computing device 304, may be embodied by one or more computing devices, processing systems, or servers, as shown in the apparatus of Figure 2.

[0088] As illustrated in Figure 2, device 200 may include processing circuitry 202, memory 204, and communications hardware 206, each of which is described in more detail below. Although various components are illustrated in Figure 2 only as connected to processing circuitry 202, device 200 may further include a bus (not explicitly shown in Figure 2) for passing information between any combination of the various components of device 200. Apparatus 200 may be configured to perform various operations related to and described above in connection with Figures 1 and 3-6 using computer program instructions.

[0089] The processing circuitry 202 (and / or any other processors assisting or associated with the coprocessor or processor) may communicate with the memory 204 via a bus for passing information between components of the apparatus. The processing circuitry 202 may be embodied in several different ways and may include, for example, one or more processing devices configured to perform independently. Additionally, the processor may include one or more processors configured in tandem via a bus to enable independent execution of software instructions, pipelining, and / or multithreading. Use of the term "processor" may be understood to include a single-core processor, a multi-core processor, multiple processors in the apparatus 200, a remote or "cloud" processor, or any combination thereof.

[0090] Processing circuitry 202 may be configured to execute software instructions stored in memory 204 or accessible to the processor (e.g., software instructions stored on another storage device (not shown)). In some cases, the processor may be configured to execute hard-coded functions. Thus, whether configured by a hardware or software method or a combination of software and hardware, processing circuitry 202, while so configured, represents an entity (an entity physically embodied in circuitry) capable of performing the operations described herein. Alternatively, as another example, if processing circuitry 202 is embodied as an executor of software instructions, the software instructions may configure processing circuitry 202 to, among other things, perform the algorithms and / or operations described herein when the software instructions are executed.

[0091] The memory 204 may be non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory 204 may be an electronic storage device (e.g., a computer-readable storage medium). The memory 204 may be configured to store information, data, content, applications, software instructions, and the like that enable the device to perform various functions in accordance with exemplary embodiments contemplated herein.

[0092] Communications hardware 206 may be any means, such as devices or circuits embodied in hardware or a combination of hardware and software, configured to receive and / or transmit data from / to a network and / or any other device, circuit, or module in communication with apparatus 200. In this regard, communications hardware 206 may include, for example, a network interface for enabling communication with a wired or wireless communication network. For example, communications hardware 206 may include one or more network interface cards, antennas, buses, switches, routers, modems, and supporting hardware and / or software, or any other devices suitable for enabling communication over a network. Additionally, communications hardware 206 may include processing circuitry for effecting transmission of such signals to a network or for manipulating receipt of signals received from a network.

[0093] Communications hardware 206 may be configured to provide output to a user, and in some embodiments, receive indicia of user input. Communications hardware 206 may include a user interface, such as a display, and may further include components that govern the use of the user interface, such as a web browser, a mobile application, a dedicated user device, etc. In some embodiments, communications hardware 206 may include a keyboard, a mouse, a touchscreen, a touch area, soft keys, a microphone, a display, a speaker, or other input, output, or presentation device. Communications hardware 206 may utilize processing circuitry 202 to control one or more functions of one or more of these user interface components via software instructions (e.g., application software and / or system software, e.g., firmware) stored in memory accessible to processing circuitry 202 (e.g., memory 204).

[0094] conclusion

[0095] Many modifications and other embodiments of the inventions described herein will suggest themselves to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing description and the associated drawings. It is therefore to be understood that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Furthermore, while the foregoing description and associated drawings describe exemplary embodiments in the context of certain illustrative combinations of elements and / or functions, it is understood that different combinations of elements and / or functions may be provided in alternative embodiments without departing from the scope of the appended claims. In this regard, for example, it is contemplated that different combinations of elements and / or functions than those explicitly set forth above may also be set forth as part of the appended claims. Where certain terms are used herein, they are used in a generic and descriptive sense and not of limitation.

Claims

1. an unmanned aerial vehicle including an imaging sensor and a radio, the radio connected to transmit real-time imaging data from the imaging sensor over a wireless network, and the radio connected to receive control data from the wireless network; a remote controller device including a radio adapted to receive real-time imaging data from the unmanned aerial vehicle over the wireless network; the remote controller device is further configured to transmit the control data to the unmanned aerial vehicle over the wireless network and to transmit a video transmission based on real-time image data received over a wide area network; a remote controller device; an end-user computing device connected to the wide area network and including one or more presentation devices, the end-user computing device configured to transmit instructions to the remote controller device to prepare the control data for transmission to the unmanned aerial vehicle; an audio guided computing device connected to the wide area network and configured to transmit supplemental audio data, the audio guided computing device configured to provide the supplemental audio data; and Including, the end-user computing devices are configured to receive the synchronized and multiplexed video transmission and the supplemental audio data over a communication channel of the wide area network, and to present the received video transmission and supplemental audio data via the one or more presentation devices; system.

2. The system of claim 1 , wherein the unmanned aerial vehicle is a quadcopter drone.

3. 10. The system of claim 1, further comprising one or more additional audio guided computing devices connected to the wide area network, each audio guided computing device providing respective supplemental audio data synchronized and multiplexed with the video transmission from the unmanned aerial vehicle.

4. 4. The system of claim 3, wherein the respective supplemental audio data transmitted by the one or more additional audio guided computing devices includes data from a local file on the additional audio guided computing device, and the transmission of the supplemental audio data is effected by the geographic location of the unmanned aerial vehicle.

5. The system of claim 4 , wherein the remote control device incorporates the voice-guided computing device.

6. The system of claim 1 , wherein the supplemental audio data is based on data related to the geographic location of the unmanned aerial vehicle.

7. The system of claim 1 , further comprising a low-orbit satellite providing a direct wireless link from the remote controller device to the unmanned aerial vehicle.

8. The system of claim 1 , further comprising a mobile communications tower providing a direct wireless link between the remote controller device and the unmanned aerial vehicle.

9. The system of claim 1 , wherein the remote controller device is configured to act as a network bridge between the wireless network and a wide area network.

10. using the real-time imaging data to identify image signatures of objects at the location of the unmanned aerial vehicle; Controlling the unmanned aerial vehicle and the imaging sensor to focus on an object having the identified image signature. The system of claim 1 , further comprising an artificial intelligence system configured to:

11. using the real-time imaging data to identify image signatures of objects at the location of the unmanned aerial vehicle; Inserting a graphic overlay onto the video transmission based on the identified object having the image signature. The system of claim 1 , further comprising an artificial intelligence system configured to:

12. using the real-time imaging data to identify image signatures of objects at the location of the unmanned aerial vehicle; and providing transmission of audio information associated with the object having the identified image signature. The system of claim 1 , further comprising an artificial intelligence system configured to:

13. The system of claim 10 , wherein the artificial intelligence system is further configured to insert a graphic overlay onto the video transmission based on the identified object having the image signature.

14. The system of claim 13 , wherein the artificial intelligence system is further configured to provide for transmission of audio information associated with the object having the identified image signature.

15. The system of claim 10 , wherein the artificial intelligence system is further configured to provide for transmission of audio information associated with the object having the identified image signature.

16. transmitting, by the unmanned aerial vehicle over a wireless network, real-time imaging data from an imaging sensor of the unmanned aerial vehicle; receiving, by the unmanned aerial vehicle, control data from the wireless network over the wireless network; receiving the real-time imaging data by a remote controller device from the wireless network; transmitting the control data by the remote controller device over the wireless network; transmitting, by an end-user computing device over a wide area network, instructions to the remote controller device to generate control data for the unmanned aerial vehicle; providing, by an audio guidance computing device, supplemental audio data; synchronizing and multiplexing a video transmission over the communication channel of the wide area network based on the real-time imaging data and the supplemental audio data for transmission to the end user computing device over the communication channel; presenting said video transmission and said supplemental audio data via one or more presentation devices of said end-user computing device; A method comprising:

17. The method of claim 16 , wherein the unmanned aerial vehicle is a quadcopter drone.

18. providing, by one or more additional audio guided computing devices, a plurality of supplemental streams of audio data for synchronization and multiplexing with the video transmission over the communication channel; The method of claim 1 further comprising:

19. The method of claim 16 , further comprising effecting transmission of the supplemental audio data according to the geographic location of the unmanned aerial vehicle.

20. The method of claim 16 , wherein the remote control device incorporates the voice-guided computing device.

21. The method of claim 16 , wherein the supplemental audio data is based on data related to the geographic location of the unmanned aerial vehicle.

22. The method of claim 16 , wherein the wireless network includes low-earth orbit satellites that provide a direct wireless link from the remote controller device to the unmanned aerial vehicle.

23. The method of claim 16 , wherein the wireless network includes a cellular tower that provides a direct wireless link between the remote controller device and the unmanned aerial vehicle.

24. The method of claim 16 , wherein the remote controller device is configured to act as a network bridge between the wireless network and a wide area network.

25. using the real-time imaging data to identify image signatures of objects at the location of the unmanned aerial vehicle; controlling the unmanned aerial vehicle and the imaging sensor to focus on an object having the identified image signature; 17. The method of claim 16, further comprising:

26. using the real-time imaging data to identify image signatures of objects at the location of the unmanned aerial vehicle; inserting a graphic overlay onto the video feed based on the identified object having the image signature; 17. The method of claim 16, further comprising:

27. using the real-time imaging data to identify image signatures of objects at the location of the unmanned aerial vehicle; causing transmission of audio information associated with the object having the identified image signature; 17. The method of claim 16, further comprising:

28. inserting a graphic overlay onto the video transmission based on the object having the identified image signature.

26. The method of claim 25, further comprising:

29. 30. The method of claim 29, further comprising the step of providing transmission of audio information associated with the object having the identified image signature.

30. 26. The method of claim 25, further comprising the step of providing transmission of audio information associated with the object having the identified image signature.

31. a large-scale language model configured to generate text describing the content of the video transmission; a speech synthesis engine configured to create speech corresponding to the created text, the supplemental audio data including the created speech; and The system of claim 1 further comprising:

32. generating a text describing the content of said video transmission using a large-scale language model; using a speech synthesis engine to generate speech corresponding to the generated text, wherein the supplemental audio data includes the generated speech; 17. The method of claim 16, further comprising:

33. 1. A voice-guided computing device for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from a remote controller device from the wireless network, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to create control data for the unmanned aerial vehicle, the remote controller device transmitting a video feed based on the real-time imaging data over the wide area network, the voice-guided computing device including a processing system including processing circuitry and a memory that stores computer program instructions, the computer program instructions comprising: providing supplemental audio data; Synchronously and multiplexing the video transmission and the supplemental audio data over a communication channel of the wide area network for transmission to the end user computing device. The audio-guided computing device is configured to: Audio-guided computing devices.

34. 1. A method for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to generate control data for the unmanned aerial vehicle, the remote controller device transmitting a video feed based on the real-time imaging data over the wide area network, the method comprising: providing supplemental audio data; synchronously and multiplexing said video transmission and said supplemental audio data over a communication channel of said wide area network for transmission to said end user computing device; A method comprising:

35. 1. A computing device for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from a remote controller device from the wireless network, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to create control data for the unmanned aerial vehicle, the remote controller device transmitting a video feed based on the real-time imaging data over the wide area network, the computing device including a processing system including processing circuitry and a memory that stores computer program instructions, the computer program instructions comprising: tracking spatial coordinates relating to the geographic location of the unmanned aerial vehicle; Generating geographic location-based information of the location of the unmanned aerial vehicle for transmission to the end user computing device. configuring a computing device so that Computing Devices

36. 1. A method for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to generate control data for the unmanned aerial vehicle, the remote controller device transmitting a video feed based on the real-time imaging data over the wide area network, the method comprising: tracking spatial coordinates relating to a geographic location of the unmanned aerial vehicle; generating geographic location-based information of the location of the unmanned aerial vehicle for transmission to the end user computing device; A method comprising:

37. an unmanned aerial vehicle including an imaging sensor and a radio, the radio connected to transmit real-time imaging data from the imaging sensor over a wireless network, and the radio connected to receive control data from the wireless network; a remote controller device including a radio adapted to receive real-time imaging data from the unmanned aerial vehicle over the wireless network; the remote controller device is further configured to transmit control data to the unmanned aerial vehicle over the wireless network and to transmit video transmissions based on real-time imaging data received over a wide area network; a remote controller device; an end-user computing device connected to the wide area network and including one or more presentation devices, the end-user computing device configured to transmit instructions to the remote controller device to prepare control data for transmission to the unmanned aerial vehicle; and connected to the wide area network, creating a representation of a three-dimensional spatial region based on a collection of polygonal prisms for defining an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence; tracking spatial coordinates relating to the geographic location of the unmanned aerial vehicle; determining a spatial relationship between the geographic location of the unmanned aerial vehicle's location and the geofence; It is configured as follows: Geofencing computing devices and Including, the system.

38. 38. The system of claim 37, wherein the unmanned aerial vehicle is a quadcopter drone.

39. 38. The system of claim 37, wherein the geofencing computing device is further configured to generate a notification to the end user computing device based on the determined spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

40. 40. The method of claim 39, wherein determining the spatial relationship includes determining whether the geographic location of the unmanned aerial vehicle is in exterior space or near the geofence.

41. To generate the notification, the geofencing computing device: storing the minimum threshold time and the minimum threshold distance; (i) the distance of the unmanned aerial vehicle to the geofence is less than or equal to the minimum threshold distance; (ii) the predicted time for the unmanned aerial vehicle to reach the geofence is less than or equal to the minimum threshold time; The notification will be issued based on either 40. The system of claim 39, configured to:

42. 42. The system of claim 41, wherein the minimum threshold time and minimum threshold distance for triggering the notification are computed values ​​based on the kinematics of the unmanned aerial vehicle.

43. 42. The system of claim 41 , wherein the minimum threshold distance is based on a maximum deceleration of the unmanned aerial vehicle in a direction perpendicular to the geofence.

44. 42. The system of claim 41 , wherein the minimum threshold time is based on a maximum deceleration of the unmanned aerial vehicle in a direction perpendicular to the geofence.

45. 40. The system of claim 39, wherein said notification comprises a graphical representation superimposed on a video transmission sent from said remote controller device to said end user computing device.

46. 40. The system of claim 39, wherein the geofencing computing device, after invoking a notification, causes the remote controller device to send control data to the unmanned aerial vehicle to decelerate away from the geofence to avoid navigating into the exterior space.

47. 38. The system of claim 37, wherein the geofencing computing device causes the remote controller device to send control data to the unmanned aerial vehicle to decelerate away from the geofence to avoid navigating into the exterior space.

48. transmitting, by the unmanned aerial vehicle, real-time imaging data from an imaging sensor on the unmanned aerial vehicle over a wireless network; receiving, by the unmanned aerial vehicle, control data from the wireless network over the wireless network; receiving the real-time imaging data by a remote controller device from the wireless network; transmitting the control data by the remote controller device over the wireless network; transmitting, by an end user computing device over a wide area network, instructions to the remote controller device to generate control data for the unmanned aerial vehicle; creating a representation of a three-dimensional spatial region based on a collection of polygonal prisms for defining an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence; tracking spatial coordinates relating to a geographic location of the unmanned aerial vehicle; determining a spatial relationship between the geographic location of the unmanned aerial vehicle's location and the geofence; A method comprising:

49. 49. The method of claim 48, wherein the unmanned aerial vehicle is a quadcopter drone.

50. 49. The method of claim 48, further comprising generating a notification to the end user computing device based on the determined spatial relationship between the geographic location of the unmanned aerial vehicle's location and the geofence.

51. 51. The method of claim 50, wherein determining the spatial relationship includes determining whether the geographic location of the unmanned aerial vehicle is in the exterior space or near the geofence.

52. the step of generating the notification comprises: storing a minimum threshold time and a minimum threshold distance; (i) the distance of the unmanned aerial vehicle to the geofence is less than or equal to the minimum threshold distance; (ii) the predicted time for the unmanned aerial vehicle to reach the geofence is less than or equal to the minimum threshold time; and invoking the notification based on any one of the following:

51. The method of claim 50, comprising:

53. 53. The method of claim 52, wherein the minimum threshold time and minimum threshold distance for eliciting the notification are computed values ​​based on the kinematics of the unmanned aerial vehicle.

54. 53. The method of claim 52, wherein the minimum threshold distance is based on a maximum deceleration of the unmanned aerial vehicle in a direction perpendicular to the geofence.

55. 53. The method of claim 52, wherein the minimum threshold time is based on a maximum deceleration of the unmanned aerial vehicle in a direction perpendicular to the geofence.

56. 51. The method of claim 50, wherein said notification comprises a graphical representation superimposed on a video transmission sent from said remote controller device to said end user computing device.

57. 51. The method of claim 50, further comprising, after invoking a notification, causing the remote controller device to send control data to the unmanned aerial vehicle to decelerate away from the geofence to avoid navigating into the exterior space.

58. 49. The system of claim 48, further comprising causing the remote controller device to send control data to the unmanned aerial vehicle to decelerate away from the geofence to avoid navigating into the exterior space.

59. 1. A geofencing computing device for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from a remote controller device from the wireless network, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to generate control data for the unmanned aerial vehicle, the remote controller device transmitting the real-time imaging data as a video transmission over the wide area network, the geofencing computing device including a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions configuring the geofencing computing device to: creating a representation of a three-dimensional spatial region based on a collection of polygonal prisms for defining an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence; tracking spatial coordinates relating to the geographic location of the unmanned aerial vehicle; determining a spatial relationship between the geographic location of the unmanned aerial vehicle's location and the geofence; Configure it as follows: Geofencing computing devices.

60. 1. A method for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to generate control data for the unmanned aerial vehicle, the remote controller device transmitting the real-time imaging data as a video transmission over the wide area network, the method comprising: creating a representation of a three-dimensional spatial region to define an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence; tracking spatial coordinates relating to a geographic location of the unmanned aerial vehicle; determining a spatial relationship between the geographic location of the unmanned aerial vehicle's location and the geofence; A method comprising:

61. 1. A geofencing computing device for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from a remote controller device from the wireless network, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to generate control data for the unmanned aerial vehicle, the remote controller device transmitting the real-time imaging data as a video transmission over the wide area network, the geofencing computing device including a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions controlling the geofencing computing device to: creating a representation of a three-dimensional spatial region defining an interior space, an exterior space, and a boundary between said interior space and said exterior space as a geofence, wherein a minimum above ground level of each point inside said geofence is determined by a visibility analysis based on three-dimensional coordinates of the geographical location of said remote controller device and an elevation map of the geographical terrain of the geofenced region; Tracking three-dimensional spatial coordinates relating to the geographic location of the unmanned aerial vehicle; determining a spatial relationship between the geographic location of the unmanned aerial vehicle's location and the geofence; Configure your computing device to geofence.

62. 1. A method for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to generate control data for the unmanned aerial vehicle, the remote controller device transmitting the real-time imaging data as a video transmission over the wide area network, the method comprising: creating a representation of a three-dimensional spatial region defining an interior space, an exterior space, and a boundary between the interior space and the exterior space as a geofence, wherein the minimum above ground level of each point inside the geofence is determined by a visibility analysis based on 1) three-dimensional coordinates of the geographic location of the remote controller device and 2) an elevation map of the geographical terrain of the geofenced region; tracking three-dimensional spatial coordinates relating to a geographic location of the unmanned aerial vehicle; determining a spatial relationship between the geographic location of the unmanned aerial vehicle's location and the geofence; A method comprising:

63. 1. A computing device for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from a remote controller device from the wireless network, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to create control data for the unmanned aerial vehicle, the remote controller device transmitting a video feed based on the real-time imaging data over the wide area network, the computing device including a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions configuring the computing device to: creating a text describing the content of the video transmission based on a large-scale language model; generating speech data corresponding to the generated text using a synthetic speech engine; Synchronously and multiplexing the video transmission and the supplemental audio data over a communication channel of the wide area network for transmission to the end user computing device. Configure your computing device to:

64. 1. A method for use in a system involving a remotely controlled unmanned aerial vehicle, the unmanned aerial vehicle including an imaging sensor and a radio that transmits real-time imaging data from the imaging sensor over a wireless network and receives control data from the wireless network from a remote controller device, the remote controller device being connected to a wide area network and receiving instructions from an end user computing device to generate control data for the unmanned aerial vehicle, the remote controller device transmitting a video feed based on the real-time imaging data over the wide area network, the method comprising: generating a text describing the content of said video transmission using a large-scale language model; generating speech data corresponding to the generated text using a synthetic speech engine; synchronously multiplexing said video transmission and said audio data over a communication channel of said wide area network for transmission to said end user computing device; A method comprising:

Images