A method and system for providing augmented reality in automobiles using unmanned aerial vehicles (UAVS)
An integrated infotainment and navigation system with a UAV addresses the limitations of existing systems by providing real-time augmented reality, enhancing driver experience and safety through comprehensive data fusion and UAV control.
Patent Information
- Application Number
- IN4691MUM2015
- Authority / Receiving Office
- IN · IN
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2015-12-14
- Publication Date
- 2019-05-17
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing in-vehicle infotainment systems and advanced driver assistance systems (ADAS) lack the ability to provide comprehensive, real-time information about surrounding areas and are constrained by line-of-sight and installation limitations, failing to meet the needs of travelers for advanced augmented reality in automobiles.
An integrated system that combines an infotainment and navigation device with a flying unmanned aerial vehicle (UAV) to provide augmented reality, allowing real-time data fusion and control of the UAV from the infotainment device, using sensors and GPS data to create composite images or videos.
Enables enhanced driver experience through hyper-reality by providing real-time, comprehensive augmented reality, including extended vision and aerial views, overcoming line-of-sight limitations and enhancing safety and information availability.
Smart Images

Figure 00000016_0000 
Figure 00000017_0000 
Figure 00000018_0000
Abstract
Description
FORM-2THE PATENT ACT,1970(39 OF 1970)ANDTHE PATENT RULES, 2003(As Amended)COMPLETE SPECIFICATION(See section 10;rule 13)" A METHOD AND A SYSTEM FOR PROVIDING AUGMENTED REALITY IN AUTOMOBILES USING UNMANNED AERIALVEHICLES (UAVS)"Tata Sons Limited, a corporation organized and existing under the laws of India, of Bombay House, 24 Homi ModyStreet, Mumbai 400 001, Maharashtra, India.The following specification particularly describes the invention and the manner in which it is to be performed:2A METHOD AND A SYSTEM FOR PROVIDING AUGMENTED REALITY INAUTOMOBILES USING UNMANNED AERIAL VEHICLES (UAVS)TECHNICAL FIELDThe present invention relates to in-vehicle infotainment and navigation system and moreparticularly, to a system and method for providing augmented reality in automobiles usingunmanned aerial vehicles (UAVs) in a real-time space.BACKGROUNDIn-Vehicle information and entertainment systems are becoming more prevalent in theautomobile industry (referred to herein as infotainment systems). Beyond radios andnavigation devices, today's vehicles are equipped with video players, game consoles, multipledisplays, external networking connections and many more.Presently, vehicle manufacturers attempt to entice travellers to use a specific conveyancebased on any number of features. Most of these features focus on-board vehicle devices suchas camera mounted over the vehicles which can guide over the road ahead. However, theylack in providing information related to the surrounding areas and any other safety-relatedconstraints in real-time. Thus, the automobile industry has failed to appease the supposedneeds of the travellers by providing more advanced infotainment systems in vehicles.Alternatively, there are several advanced driver assistance systems (ADAS), equipped withcameras, radar / lidar or any other device. ADAS provides real-time information related tosurrounding environments. However, these ADAS are also constrained by the line-of-sightand other installation limitations.Hence, there is a need to have a system and method that can overcome the above statedproblems and provides a system and method for augmented reality in automobiles.SUMMARYThe following presents a simplified summary of the subject matter in order to provide a basicunderstanding of some aspects of subject matter embodiments. This summary is not an3extensive overview of the subject matter. It is not intended to identify key / critical elements ofthe embodiments or to delineate the scope of the subject matter.Its sole purpose is to present some concepts of the subject matter in a simplified form as aprelude to the more detailed description that is presented later.It is therefore a primary objective of this invention to provide a system and method foraugmented reality in automobiles.According to the preferred embodiment, an integrated in-vehicle system which comprises aninfotainment and navigation device operatively connected with at least a flying unmannedaerial vehicles (UAVs). Said flying unmanned aerial vehicles (UAVs) are capable ofproviding augmented reality information related to surrounding areas as well as the roadahead. Further, the movement of said flying unmanned aerial vehicles (UAVs) can becontrolled from the infotainment and navigation device.In another embodiment, the present invention provides a system (100) for providingaugmented reality, said system comprising an automobile (101) comprising an infotainmentand navigation device (102); and an unmanned aerial vehicle (103), operatively coupled tosaid automobile, to provide real-time data exterior to the automobile; said infotainment andnavigation device (102), operable to create and display real-time composite images or videos,by combining at least part of said real-time data obtained from the unmanned aerial vehiclewith data available internally.In another embodiment, the composite images or videos are created, in the infotainment andnavigation device, by superimposing or overlaying video or images as obtained in real timefrom the unmanned aerial vehicle with GPS data available from an internal navigation device.In another embodiment, the unmanned aerial vehicle comprises a plurality of sensors toobtain said real-time data.In another embodiment, the automobile comprises a plurality of control elements, operable tocontrol the unmanned aerial vehicle and the infotainment and navigation device. The controlelements are placed on a steering wheel of the automobile.4In another embodiment, the infotainment and navigation device comprises a display, aprocessor and a memory. The display is coupled to the processor and the memory.In another embodiment, the present invention a method for providing an augmented reality,said method comprising obtaining, external real-time data using an Unmanned Aerial Vehicle(UAV); providing said data to an infotainment and navigation device of the automobile; andcreating and displaying, composite images or videos by combining said external real-timedata with internal data.These and other objects, embodiments and advantages of the present invention will becomereadily apparent to those skilled in the art from the following detailed description of theembodiments having reference to the attached figures, the invention not being limited to anyparticular embodiments disclosed.Brief Description of the DrawingsFor a better understanding of the embodiments of the systems and methods described herein,and to show more clearly how they may be carried into effect, reference will now be made,by way of example, to the accompanying drawings, wherein:FIGURE 1 illustrates a graphic representation of a system for providing augmentedreality in accordance with the present invention.FIGURE 2 illustrates a block diagram of an internal configuration of the infotainment andnavigation device and the UAV in accordance with the present invention.FIGURE 3 illustrates a method for providing augmented reality in accordance with thepresent invention.DESCRIPTIONExemplary embodiments now will be described with reference to the accompanyingdrawings. The invention may, however, be embodied in many different forms and should notbe construed as limited to the embodiments set forth herein; rather, these embodiments are5provided so that this invention will be thorough and complete, and will fully convey its scopeto those skilled in the art. The terminology used in the detailed description of the particularexemplary embodiments illustrated in the accompanying drawings is not intended to belimiting. In the drawings, like numbers refer to like elements.The specification may refer to “an”, “one” or “some” embodiment(s) in several locations.This does not necessarily imply that each such reference is to the same embodiment(s), orthat the feature only applies to a single embodiment. Single features of differentembodiments may also be combined to provide other embodiments.As used herein, the singular forms “a”, “an” and “the” are intended to include the pluralforms as well, unless expressly stated otherwise. It will be further understood that the terms“includes”, “comprises”, “including” and / or “comprising” when used in this specification,specify the presence of stated features, integers, steps, operations, elements, and / orcomponents, but do not preclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and / or groups thereof. It will beunderstood that when an element is referred to as being “connected” or “coupled” to anotherelement, it can be directly connected or coupled to the other element or intervening elementsmay be present. Furthermore, “connected” or “coupled” as used herein may includeoperatively connected or coupled. As used herein, the term “and / or” includes any and allcombinations and arrangements of one or more of the associated listed items.Unless otherwise defined, all terms (including technical and scientific terms) used hereinhave the same meaning as commonly understood by one of ordinary skill in the art to whichthis invention pertains. It will be further understood that terms, such as those defined incommonly used dictionaries, should be interpreted as having a meaning that is consistentwith their meaning in the context of the relevant art and will not be interpreted in an idealizedor overly formal sense unless expressly so defined herein.The figures depict a simplified structure only showing some elements and functional entities,all being logical units whose implementation may differ from what is shown. Theconnections shown are logical connections; the actual physical connections may be different.It is apparent to a person skilled in the art that the structure may also comprise otherfunctions and structures.6Also, all logical units described and depicted in the figures include the software and / orhardware components required for the unit to function. Further, each unit may comprisewithin itself one or more components which are implicitly understood. These componentsmay be operatively coupled to each other and be configured to communicate with each otherto perform the function of the said unit.The detailed description follows in parts to terms of processes and symbolic representations ofoperations performed by conventional computers, including computer components. For thepurpose of this invention, a computer may be any microprocessor or processor (hereinafterreferred to as processor) controlled device such as, by way of example, personal computers,workstations, servers, clients, minicomputers, main-frame computers, laptop computers, anetwork of one or more computers, mobile computers, portable computers, handheldcomputers, palm top computers, set-top boxes for a TV, interactive televisions, interactivekiosks, personal digital assistants, interactive wireless devices, mobile browsers, or anycombination thereof.For the most part, the operations described herein are operations performed by a computer ora machine in conjunction with a human operator or user that interacts with the computer or themachine. The programs, modules, processes, methods, and the like, described herein are butan exemplary implementation and are not related, or limited, to any particular computer,apparatus, or computer language. Rather, various types of general purpose computingmachines or devices may be used with programs constructed in accordance with the teachingsdescribed herein.It would be well appreciated by persons skilled in the art that the term “module” and “unit”can be interchangeably used in the present invention.It would be well appreciated by persons skilled in the art that the term “unmanned aerialvehicle (UAV)” and “drone” can be interchangeably used in the present invention.The present invention describes a unique way in which hardware and software components ofIn-Vehicle Infotainment / Human Machine Interface (IVI / HMI) and flying Drones or UnmannedAerial Vehicles are integrated in a unified way for enhancing a driver’s experience for differentuse cases involving motoring. Further, the integrated capabilities of drone systems and7vehicular IVI / HMI systems offers a synergistic effect, not possible by the individual features,thereby providing unique experiences to the driver.Figure 1 illustrates a graphic representation of a system for providing augmented reality inaccordance with the present invention. The system (100) comprising an automobile (101)with an infotainment and navigation device (102). It further comprises a flying unmannedaerial vehicle (UAV) (103) which is wirelessly connected with the automobile and theinfotainment and navigation device. Further, the operation and control of the flying UAV ispossible with the dashboard of the infotainment and navigation system.Additionally, the UAV is mounted with plurality of sensors to obtain various types of data insurrounding environment.Figure 2 illustrates a block diagram of an internal configuration of the infotainment andnavigation device and the UAV in accordance with the present invention. The Unmannedaerial vehicle comprises an on-board control unit (201). The on-board control unit (201) ofthe flying UAV comprises a processor (201a) coupled with a memory (201b) to perform all thecontrol operations and a communication unit or a transceiver (201c) to wirelessly communicatewith transceiver (202a) of the infotainment and the navigation device. Similarly, theinfotainment and navigation device comprises an integrated processor (202b) coupled with amemory (202c), a display (203) and a dashboard (204). It would be well appreciated bypersons skilled in the art that the display / dashboard can be a touchscreen. Further, the controlof the UAV could be managed via steering mounted controls connected with infotainmentsystem via a communication bus line. The driver or co-passenger can control the UAV eitherthrough the touch-screen or steering-wheel-mounted elements.Further, the communication of control and data signal from the moving automobile on aground to the flying unmanned aerial vehicle (UAV) involves sending accurate vehicleposition with a much powerful GPS device fitted to the UAV for better alignment of GPScoordinates in “follow me” or “look ahead” mode.It will be well understood by a person skilled in the art that integrated vehicle and UAVsystems would allow for controlling of an aerial object (an UAV) by a moving object on theground (an automobile) autonomously.8In another embodiment, the UAV control unit may also communicate with one or moresensors, which are either associated with the UAV or communicate with the UAV. UAVsensors may include one or more sensors for providing information to the vehicle controlunit that determine or provide information about the environment in which the vehicle isoperating. Examples of information provided by the sensors and that may be used by theUAV control unit may include natural environment, terrain data, weather tracking data,traffic data, user health tracking data, or other types of data, which may provideenvironmental or other data to the UAV control unit. The UAV control unit may also performsignal processing of signals received from one or more sensors. Such signal processing mayinclude estimation of a measured parameter from a single sensor, such as multiplemeasurements of a range state parameter Signal processing of such sensor signalmeasurements may comprise stochastic signal processing, adaptive signal processing, and / orother signal processing techniques known to those skilled in the art.Further, the sensors operates in an area what can be detected by those sensors associated withthe UAV. Although sensor range is shown as a fixed and continuous, the sensor range may bedynamic and / or discontinuous. For example, a camera may provide real-time images of thesurrounding images, or an environmental sensor may provide environmental conditions (e.g.,rain, fog, clear, etc.). Thus, the environment may have an area that includes all areas withinthe sensor range. The area may include locations of travel that the vehicle may proceed to inthe future.The integrated vehicle and UAV systems would allow vehicular dynamic data to be sharedwith the on-board control unit of the UAV through the infotainment and navigation device. Inanother embodiments, as the vehicle speed increases from stop position to a moving position,the on-board control unit of the UAV is automatic signalled to take-off. As the vehicle’sspeed increases, the on-board control unit can adjust the speed of the flying UAV, and whenthe vehicle stops and engine is turned-off, the on-board control unit can signal the flyingUAV to land on the stationary vehicle. This would also allow for taking off / landing of theUAV from / on a moving automobile.9In another embodiments, infotainment and navigation device is capable of capturing othernavigation information by another like means and can transfer that data to the UAV via RFcommunication linkFigure 3 illustrates a method for providing augmented reality in accordance with the presentinvention. In step (301), obtaining, external real-time data using an Unmanned Aerial Vehicle(UAV). In step 302, providing said data to an infotainment and navigation device of theautomobile. In step (303), creating and displaying, composite images or videos by combiningsaid external real-time data with internal data.Further, said creating of composite images or videos, in the infotainment and navigationdevice, is by superimposing or overlaying video or images, as obtained in real time from theunmanned aerial vehicle with GPS data available from an internal navigation device.In an exemplary embodiment, the augmentation would be to superimpose or overlay livevideo or composite images transmitted from the UAV onto a 3D-navigation system.According to requirement of the driver, the view-angle and height of the 3D-view of thestreets / buildings shown in the navigation are determined and then these get translated foradjusting the viewing angle and height of the UAV to capture real-time video and transmittedat the same rate as the displacement of the map on the screen so that video overlay on the 3Dviewmap will be proper.The term “bus” and variations thereof, as used herein, can refer to a subsystem that transfersinformation and / or data between various components. A bus generally refers to the collectioncommunication hardware interface, interconnects, bus architecture, standard, and / or protocoldefining the communication scheme for a communication system and / or communicationnetwork. A bus may also refer to a part of a communication hardware that interfaces thecommunication hardware with the interconnects that connect to other components of thecorresponding communication network. The bus may be for a wired network, such as aphysical bus, or wireless network, such as part of an antenna or hardware that couples thecommunication hardware with the antenna. A bus architecture supports a defined format in10which information and / or data is arranged when sent and received through a communicationnetwork. A protocol may define the format and rules of communication of a bus architecture.The term “communication unit” or “transceiver” and variations thereof, as used herein, canrefer to a collection of communication components capable of one or more of transmission,relay, interconnect, control, or otherwise manipulate information or data from at least onetransmitter to at least one receiver. As such, the communication may include a range ofsystems supporting point-to-point or broadcasting of the information or data. Acommunication system may refer to the collection individual communication hardware aswell as the interconnects associated with and connecting the individual communicationhardware. Communication hardware may refer to dedicated communication hardware or mayrefer a processor coupled with a communication means (i.e., an antenna) and runningsoftware capable of using the communication means to send and / or receive a signal withinthe communication system. Interconnect refers some type of wired or wirelesscommunication link that connects various components, such as communication hardware,within a communication system. A communication network may refer to a specific setup of acommunication system with the collection of individual communication hardware andinterconnects having some definable network topography. A communication network mayinclude wired and / or wireless network having a pre-set to an ad hoc network structure.The terms dash and dashboard and variations thereof, as used herein, may be usedinterchangeably and can be any panel and / or area of a vehicle disposed adjacent to anoperator, user, and / or passenger. Dashboards may include, but are not limited to, one or morecontrol panel(s), instrument housing(s), head unit(s), indicator(s), gauge(s), meter(s), light(s),audio equipment, computer(s), screen(s), display(s), HUD unit(s), and graphical userinterface(s).The term “display” refers to a portion of a physical screen used to display the output to auser. Such as console display, instrument cluster, heads-up display.The term “GPS” refers to the Global Positioning System in which satellite-based radiosystems, typically satellite-based, provide three-dimensional position and time information towhich suitably equipped receivers anywhere on or near the surface of the Earth compute theirglobal position, velocity and orientation (heading).11The terms “infotainment” and “infotainment device” may be used interchangeably and canrefer to the hardware / software products, data, content, information, and / or systems, whichcan be built into or added to vehicles to enhance driver and / or passenger experience.Infotainment may provide media and / or multimedia content. An example is informationbasedmedia content or programming that also includes entertainment content.The term “screen,” “touch screen,” “touchscreen,” or “touch-sensitive display” refers to aphysical structure that enables the user to interact with the computer by touching areas on thescreen and provides information to a user through a display. The touch screen may sense usercontact in a number of different ways, such as by a change in an electrical parameter (e.g.,resistance or capacitance), acoustic wave variations, infrared radiation proximity detection,light variation detection, and the like. In a resistive touch screen, for example, normallyseparated conductive and resistive metallic layers in the screen pass an electrical current.When a user touches the screen, the two layers make contact in the contacted location,whereby a change in electrical field is noted and the coordinates of the contacted locationcalculated. In a capacitive touch screen, a capacitive layer stores electrical charge, which isdischarged to the user upon contact with the touch screen, causing a decrease in the charge ofthe capacitive layer. The decrease is measured, and the contacted location coordinatesdetermined. In a surface acoustic wave touch screen, an acoustic wave is transmitted throughthe screen, and the acoustic wave is disturbed by user contact. A receiving transducer detectsthe user contact instance and determines the contacted location coordinates.The terms “vehicle,” “car,” “automobile,” and variations thereof may be usedinterchangeably herein and can refer to a device or structure for transporting animate and / orinanimate or tangible objects (e.g., persons and / or things), such as a self-propelledconveyance. A vehicle as used herein can include any conveyance or model of a conveyance,where the conveyance was originally designed for the purpose of moving one or moretangible objects, such as people, animals, cargo, and the like. The term “vehicle” does notrequire that a conveyance moves or is capable of movement. Typical vehicles may includebut are in no way limited to cars, trucks, motorcycles, busses, automobiles, trains, railedconveyances, boats, ships, marine conveyances, submarine conveyances, airplanes, spacecraft, flying machines, human-powered conveyances, and the like.12According to an embodiment of the present invention, the user has been given a choice toexperience “hyper-reality” by engaging functionalities to view images / videos transmitted froma camera mounted on the flying UAV above the vehicle. The driver also has the options ofcontrolling manoeuvres of the UAV and also the movements of the camera and other sensorsmounted on the drone. The video / image will be shown on any of the display. Other applicationsfor enhanced driver’s experience utilizing images / videos captured for surveillance, trafficmanagement or any other like data.In another embodiment, on-board cameras on UAV also provide 360view or surround viewbased system which gives a perspective of nearby spaces around the vehicle. This view can beaugmented with the view of a UAV hovering on the top of the vehicle thereby broadening theview / perspective of the driver for the vehicle.In an advantageous embodiment, the present invention provides a new driving experience of“hyper-reality” by feeding aerial view or extended vision of the scene ahead with the help ofa drone / UAV.The present invention is applicable to all types of on-chip and off chip memories used invarious in digital electronic circuitry, or in hardware, firmware, or in computer hardware,firmware, software, or in combination thereof. Apparatus of the invention can beimplemented in a computer program product tangibly embodied in a machine-readablestorage device for execution by a programmable processor; and methods actions can beperformed by a programmable processor executing a program of instructions to performfunctions of the invention by operating on input data and generating output. The inventioncan be implemented advantageously on a programmable system including at least one inputdevice, and at least one output device. Each computer program can be implemented in a highlevelprocedural or object-oriented programming language or in assembly or machinelanguage, if desired; and in any case, the language can be a compiled or interpreted language.Suitable processors include, by way of example, both general and specific microprocessors.Generally, a processor will receive instructions and data from a read-only memory and / or arandom access memory. Generally, a computer will include one or more mass storage devicesfor storing data file; such devices include magnetic disks and cards, such as internal hard13disks, and removable disks and cards; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructions and data include allforms of volatile and non-volatile memory, including by way of example semiconductormemory devices, such as EPROM, EEPROM, and flash memory devices; magnetic diskssuch as internal hard disks and removable disks; magneto-optical disks; CD-ROM and DVDROMdisks; and buffer circuits such as latches and / or flip flops. Any of the foregoing can besupplemented by, or incorporated in ASICs (application-specific integrated circuits), FPGAs(field-programmable gate arrays) and / or DSPs) digital signal processors).It will be apparent to those having ordinary skill in this art that various modifications andvariations may be made to the embodiments disclosed herein, consistent with the presentinvention, without departing from the spirit and scope of the present invention. Otherembodiments consistent with the present invention will become apparent from considerationof the specification and the practice of the description disclosed herein.14
Claims
We claim:
1. A system (100) for providing augmented reality, said system comprising:an automobile (101) comprising an infotainment and navigation device (102); andan unmanned aerial vehicle (103), operatively coupled to said automobile, to providereal-time data exterior to the automobile;said infotainment and navigation device (102), operable to create and display realtimecomposite images or videos, by combining at least part of said real-time dataobtained from the unmanned aerial vehicle with data available internally.
2. The system as claimed in claim 1, wherein said unmanned aerial vehicle comprises aplurality of sensors to obtain said real-time data.
3. The system as claimed in claim 1, wherein said automobile comprises a plurality ofcontrol elements, operable to control the unmanned aerial vehicle and theinfotainment and navigation device.
4. The system as claimed in claim 3, wherein said control elements are located on asteering wheel of the automobile.
5. The system as claimed in claim 1, wherein said composite images or videos arecreated, in the infotainment and navigation device, by superimposing or overlayingvideo or images as obtained in real time from the unmanned aerial vehicle with GPSdata available from an internal navigation device.
6. The system as claimed in claim 1, wherein said data is an image or video data ofsurrounding natural environment, terrain, weather or traffic.
7. A method for providing augmented reality in an automobile, said method comprising:obtaining, external real-time data using an Unmanned Aerial Vehicle (UAV);15providing said data to an infotainment and navigation device of the automobile; andcreating and displaying, composite images or videos by combining said external realtimedata with internal data.
8. The method as claimed in claim 7, wherein the method comprises obtaining data froma plurality of sensors disposed on the unmanned aerial vehicle.
9. The method as claimed in claim 7, wherein the method comprises controlling theunmanned aerial vehicle and the infotainment and navigation device using a pluralityof control elements.
10. The method as claimed in claim 7, wherein said creating of composite images orvideos, in the infotainment and navigation device, is by superimposing or overlayingvideo or images, as obtained in real time from the unmanned aerial vehicle with GPS
