Vehicle cabin learning experience using augmented reality

Abstract

The present disclosure provides systems and methods for allowing a passenger of a vehicle to interact with an ambient environment in an educational and entertaining manner using an augmented reality (AR) touch screen window. The passenger may select an element of the ambient environment by touching a location of the element on the AR window. An image of the element may be captured and displayed in a portion of the AR window along with information about the element retrieved from the Internet. The passenger may use the touchscreen of the AR window to select the image to view an Internet browser with additional information about the selected element. Additionally, the passenger may edit the information and / or add additional annotations, add additional images, audio and / or video clips, etc. to generate mixed media content, which may be shared or stored in a cloud-based storage device for later viewing.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to augmented reality and virtual reality presentations generated for occupants of a vehicle. Background Art

[0002] Traffic jams and long vehicle journeys may make the vehicle's occupants experience dull and boring due to the lack of interactive activities within the vehicle. Occupants may rely on the use of various gadgets such as tablets, smart phones, etc. for entertainment, which may hinder the occupants' health, especially for children. Occupants may see interesting elements of the surrounding environment, but there is currently no way to learn more about these elements in a short period of time (e.g., when passing by in the vehicle). Occupants may see destinations that they are interested in visiting and may want to create a travel plan to the destination. However, planning a trip to a destination may require searching on the Internet or viewing a physical travel guide, which may be cumbersome while riding in a vehicle. Summary of the invention

[0003] The present disclosure at least partially solves one or more of the above problems through a method for a computing system for a vehicle, the method comprising: in response to a passenger of the vehicle selecting an element of the surrounding environment of the vehicle, the element being visible to the passenger through a window of the vehicle when the vehicle is moving: capturing a first image of the surrounding environment including the element via one or more cameras of the vehicle; identifying the element of the first image; retrieving information about the element from the Internet; and displaying a second image of the element and the retrieved information about the element on the window. The window may be an augmented reality (AR) window including a touch screen portion, wherein the passenger may select the element by touching a location of the element on the AR window. After the second image is displayed on the window, the passenger may select the second image using the touch screen to view an Internet browser with additional information (e.g., search results) about the selected element. In addition, the passenger may edit the information and / or add additional annotations, attach additional images, audio, and / or video clips to the second image. The second image and associated content may be stored in a database of the vehicle and / or in a cloud-based storage device for later viewing. The second image and associated content may also be shared with other passengers of the vehicle. For example, a first passenger may request that the second image and associated content be sent to a second passenger. The computing system may display the request on a second AR window of the second passenger. The second passenger may accept the request, and the second image and associated content may be displayed on the second AR window.

[0004] In this way, passengers of the vehicle can interact with the surrounding environment in an entertaining and educational way. Children can learn about things they see during their journey in the vehicle and share what they learn with others. Adults can plan future journeys and save plans in a customized trip planner for future reference. Passengers of all ages can record their journey in a multimedia travel log or album, saving images and information collected along the journey for later viewing and / or sharing.

[0005] The above advantages and other advantages and features of the present description will be readily apparent from the following detailed description and understood individually or in conjunction with the accompanying drawings. It should be understood that the above summary of the invention is provided to introduce a series of concepts further described in the detailed description in a simplified form. This is not meant to identify the key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims attached to the detailed description. In addition, the claimed subject matter is not limited to implementations that address any shortcomings noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the accompanying drawings, in which:

[0007] Figure 1 is a schematic diagram of a vehicle according to one or more embodiments of the present disclosure;

[0008] Figure 2 A block diagram illustrating an exemplary embodiment of a vehicle computing system according to one or more embodiments of the present disclosure;

[0009] Figure 3 is a flow chart illustrating an exemplary high-level method for processing visual data captured by one or more cameras of a vehicle according to one or more embodiments of the present disclosure;

[0010] Figure 4 is a flow chart illustrating an exemplary method for displaying data related to a surrounding environment captured by one or more cameras of a vehicle on a window of the vehicle according to one or more embodiments of the present disclosure;

[0011] Figure 5 is a first example of AR display of information related to a surrounding environment displayed on a window of a vehicle according to one or more embodiments of the present disclosure;

[0012] Figure 6 is a second example of AR display of information related to a surrounding environment displayed on a window of a vehicle according to one or more embodiments of the present disclosure; and

[0013] Figure 7is a third example of AR display of information related to surrounding environment displayed on a window of a vehicle according to one or more embodiments of the present disclosure.

[0014] The accompanying drawings illustrate specific aspects of the described systems and methods. Together with the following description, the accompanying drawings demonstrate and explain the structures, methods, and principles described herein. In the accompanying drawings, the sizes of components may be exaggerated or otherwise modified for clarity. Well-known structures, materials, or operations are not shown or described in detail to avoid obscuring various aspects of the described components, systems, and methods. DETAILED DESCRIPTION

[0015] Vehicles may include various systems for monitoring the internal and external environment of the vehicle and conveying information to the driver of the vehicle based on the monitored internal and external environment. As an example, advanced driver assistance systems (ADAS) are developed to adapt and enhance the operation of the vehicle, thereby improving the driving quality and driving experience. ADAS uses various types of sensors such as cameras, lidar, radar, and ultrasonic sensors to monitor the internal and external environment of the vehicle. The sensor data can be processed by an on-board computing system to extract information about the vehicle operator / occupant, and convey the information to the operator and / or implement actions based on the information. These actions may include warning the operator via audio or visual notifications, or controlling the throttle, brakes, or steering of the vehicle.

[0016] ADAS can rely on a driver monitoring system (DMS) and / or an occupant monitoring system (OMS). DMS includes a set of advanced safety features that apply outputs from in-cabin sensors (including cameras) to detect and track the operator / driver's physical state (e.g., the vehicle operator experiences drowsiness) or mental state (e.g., the vehicle operator is distracted). In addition, DMS can warn the vehicle operator when the vehicle operator is experiencing a physical state or mental state that is not suitable for vehicle operation. Compared with DMS, OMS can monitor the entire in-cabin environment. OMS includes features such as seat belt detection, smart airbag deployment, left object warning, and forgotten child warning.

[0017] ADAS, DMS and OMS systems are generally operator-oriented (e.g., driver) usage information, for example, to increase the safety of the operator and occupants of the vehicle and / or to increase the driving performance of the operator. ADAS, DMS and OMS systems may not provide information to passengers of the vehicle. In some cases, rear seat passengers of the vehicle may access information for educational or entertainment purposes via a rear seat entertainment system (RSE), where the rear seat passengers may interact with the touch screen of the RSE to view video content, browse the Internet, etc. However, the RSE system may not be integrated with the ADAS, DMS, OMS and / or interior / exterior cameras of the vehicle, where video data captured by the interior / exterior cameras and / or processed by the ADAS, DMS and OMS systems are not available via the RSE system.

[0018] Information about the surroundings may be of interest to vehicle passengers for a variety of reasons. During a journey, particularly a long journey or in high congestion conditions, a passenger may look out the vehicle's windows and may wish to learn more about elements of the surroundings, such as geographic features, natural parks, historical buildings, museums, etc. For example, a passenger may see a mountain range and may wish to know the name, height, or other data of the mountain range; a passenger may see a tourist attraction and may wish to know information about visiting the tourist attraction; a passenger may see a monument and may wish to learn about the history of the monument; and so on.

[0019] Currently, passengers can access such information via personal computing devices such as smart phones or tablet computers. In some embodiments, such information may be available to passengers via an RSE in addition or alternatively. However, accessing the information may include performing a search for elements of the surrounding environment in an Internet browser. Performing the search may require manually entering information via a touch screen of a device or RSE; waiting for a response; searching for links returned by the search and filtering information; reading the entries to determine the suitability or desirability of the information; and the like. As a result, obtaining information about elements of the surrounding environment may be cumbersome and / or time consuming.

[0020] Additionally, when the vehicle is moving, retrieving information about the element may be difficult. By the time the information is retrieved, the element of the surrounding environment may no longer be visible through the window, and the passenger may lose interest. Furthermore, when the passenger is a child, the child may not have adequate education or skills to retrieve the required information. Furthermore, using a personal computing device may require looking away from the window, and excessive use of a handheld device may be unhealthy for the passenger's posture, vision, or mental state.

[0021] To address this problem, systems and methods for an in-vehicle augmented reality (AR) system are proposed herein, which process image data captured at one or more cameras (e.g., external cameras) of a vehicle and display information about elements of the surrounding environment on the surface of a window of the vehicle. The information can be generated automatically or based on user interaction. For example, in various embodiments, the window can include a touch screen portion, and a passenger can select a feature in the surrounding environment by touching the position of the feature on the touch screen portion (e.g., the surface of the window). With the aid of one or more machine learning (ML) models, the AR system can process image data from one or more cameras to determine the selected feature, retrieve information related to the selected feature from the Internet, filter the information, and display (e.g., project) the information on the surface of the window. The displayed information can be positioned or arranged on the surface of the window so that the information can be viewed simultaneously with the features of the environment. The displayed information can be generated and displayed in real time.

[0022] In this way, riding in a vehicle can be transformed into a learning experience in which passengers can learn about elements of the surrounding environment in a fun, easy, gamified manner. Additionally, in various embodiments, passengers can interact with the displayed information by adding relevant information, editing the information, and / or sharing the information with other passengers of the vehicle.

[0023] Figure 1 An exemplary vehicle is shown in FIG. Figure 2 A schematic diagram of a computing system of a vehicle including an in-vehicle AR system is shown in FIG. The occupants of the vehicle can interact with the in-vehicle AR system by touching the AR touch screen window, such as Figure 5 In response to the occupant selecting an element of the vehicle's surroundings via the AR touchscreen window, the computing system may retrieve images from multiple cameras of the vehicle and, by following Figure 3 The images are processed by one or more steps of the method to determine whether the selected element is a high priority element related to operating the vehicle. If the selected element is a low priority feature (e.g., not a high priority feature), the image may be processed by following Figure 4 One or more steps of the method are used to generate educational or entertainment information related to the selected element and display it to the passenger. The educational or entertainment information related to the selected element can be projected on the surface of the window of the vehicle, such as Figure 6 Passengers can manipulate, learn more, remove and / or share educational or entertainment information via the AR touch screen window, such as Figure 7 shown.

[0024] Now turning to the attached figure, Figure 1An exemplary vehicle 100 is schematically shown. The vehicle 100 includes an instrument panel 102, a driver's seat 104, a first passenger seat 106, a second passenger seat 108, and a third passenger seat 110. In other examples, the vehicle 100 may include more or fewer passenger seats. The driver's seat 104 and the first passenger seat 106 are located at the front of the vehicle, near the instrument panel 102, and thus may be referred to as front seats. The second passenger seat 108 and the third passenger seat 110 are located at the rear of the vehicle and may be referred to as rear seats (or rear seats).

[0025] The vehicle 100 includes a plurality of integrated speakers 114, which may be arranged around the periphery of the vehicle 100. In some embodiments, the integrated speakers 114 are electronically coupled to the electronic control system of the vehicle, such as the computing system 120, via a wired connection. In other embodiments, the integrated speakers 114 may communicate wirelessly with the computing system 120. As an example, an audio file may be generated by the computing system 120 or selected by an occupant of the vehicle 100, and the selected audio file may be played at one or more integrated speakers 114. In some examples, an audio alert may be generated by the computing system 120 and may also be played at the integrated speakers 114. In some embodiments, an audio file, signal, and / or alert may be selected by or generated for an occupant of the vehicle 100 and may be played at an integrated speaker 114 associated with and / or proximate to the occupant's seat. Additionally, in some embodiments, an audio file, signal, and / or alert selected by or generated for an occupant may be played at an integrated speaker 114 associated with and / or proximate to the seat of one or more other occupants of the vehicle 100.

[0026] The vehicle 100 includes a steering wheel 112 and a steering column 122 through which a driver can input steering instructions for the vehicle 100. The vehicle 100 also includes one or more cameras 118, which can be directed at respective occupants of the vehicle. Figure 1 In the illustrated embodiment, the camera 118 is positioned to the side of the driver's seat 104, which can help monitor the driver's profile. However, in other examples, one or more cameras 118 can be positioned at other locations in the vehicle, such as on the steering column 122, in front of the driver's seat 104, and / or to the side or in front of the passenger seats 106, 108, and 110. The one or more cameras 118 can be arranged so that the one or more cameras 118 collectively capture all occupants.

[0027] In addition, Figure 1In the illustrated embodiment, one or more cameras 119 may be positioned on the exterior of the vehicle 100 which may capture images of the surroundings of the vehicle 100 and help monitor the position of the vehicle 100 in the lane and / or monitor the position of the vehicle 100 relative to other vehicles and / or the surroundings. Figure 1 100, but it should be understood that one or more cameras 119 may be positioned at other locations on the exterior of the vehicle 100. For example, a first camera 119 may be positioned at the rear end 170 of the vehicle 100; a second camera 119 may be positioned at the front end 172 of the vehicle 100; a third camera 119 may be positioned at the right side 174 of the vehicle 100; and a fourth camera 119 may be positioned at the left side 176 of the vehicle 100. In some examples, the camera 119 may be located in a side mirror of the vehicle 100. In some embodiments, a plurality of cameras 119 may be arranged on each of one or more sides of the vehicle 100. Additionally or alternatively, in some embodiments, one or more cameras 119 may be positioned at different locations within the vehicle 100 and directed outward from the vehicle 100 toward the surrounding environment, for example, through different windows of the vehicle. The one or more cameras 119 may be arranged so that the one or more cameras 119 collectively capture a 360-degree view of the surrounding environment.

[0028] The vehicle 100 may also include a driver seat sensor 124 coupled to or within the driver seat 104 and a passenger seat sensor 126 coupled to or within the first passenger seat 106. The rear seats may also include seat sensors, such as a passenger seat sensor 128 coupled to the second passenger seat 108 and a passenger seat sensor 130 coupled to the third passenger seat 110. The driver seat sensor 124 and the passenger seat sensor 126 may each include one or more sensors, such as a weight sensor, a pressure sensor, and one or more seat position sensors, which output measurement signals to the computing system 120. For example, the computing system 120 may use the output of the weight sensor or the pressure sensor to determine whether the corresponding seat is occupied, and if occupied, determine the weight of the person occupying the seat. As another example, the computing system 120 may use the output of one or more seat position sensors to determine one or more of the occupants of the vehicle 100, the seat height, the longitudinal position relative to the dashboard 102 and the rear seats, and the angle (e.g., tilt) of the seat back of the corresponding seat.

[0029] In some examples, the vehicle 100 also includes a driver seat motor 134 coupled to or positioned in the driver seat 104 and a passenger seat motor 138 coupled to or positioned in the first passenger seat 106. Although not shown, in some embodiments, the rear seat may also include a seat motor. The driver seat motor 134 may be used to adjust the seat position, including seat height, longitudinal seat position, and the angle of the seat back of the driver seat 104, and may include an adjustment input 136. For example, the adjustment input 136 may include one or more toggle switches, buttons, and switches. The passenger seat motor 138 may be used to adjust the seat position, including seat height, longitudinal seat position, and the angle of the seat back of the first passenger seat 106, and may include an adjustment input 140. The adjustment input 140 may include one or more toggle switches, buttons, and switches. Although not shown, in some embodiments, the rear seat may be adjusted in a similar manner.

[0030] The computing system 120 may include a user interface (UI) 116. The computing system 120 may receive input via the UI 116 and output information to the UI 116. The user interface 116 may be included in, for example, a digital cockpit and may include a display and one or more input devices. The one or more input devices may include one or more touch screens, knobs, dials, hard buttons, and soft buttons for receiving user input from a vehicle occupant. The UI 116 may include a display screen on which information, images, videos, etc. may be displayed. The display screen may be a touch screen, and the occupants of the vehicle 100 may interact with the UI 116 via control elements displayed on the display screen. In some embodiments, the UI 116 may additionally or alternatively display or project information, images, videos, etc. on the windshield 113 of the vehicle 100. In addition, in some embodiments, different visual content may be generated and displayed on different display screens of the dashboard or at different locations of the windshield 113, so that the driver may view a first set of visual content and the front seat passenger of the vehicle 100 may view a different second set of visual content. For example, a first set of visual content may be related to operating a vehicle, while a second set of visual content may not be related to operating a vehicle.

[0031] The computing system 120 may include one or more additional UIs 117 that may be located in a position accessible to rear seat passengers of the vehicle 100. In various embodiments, one or more additional UIs 117 may include or be composed of an RSE system. In other words, the first UI 117 may be located behind the front seat 106 so that a first rear passenger sitting in the rear seat 108 can interact with the first UI 117. The second UI 117 may be located behind the driver's seat 104 so that a second rear passenger sitting in the rear seat 110 can interact with the second UI 117. In a vehicle that includes additional rear seats, the additional UI 117 may be similarly arranged so that each occupant of the vehicle 100 can interact with the UI 117.

[0032] Each additional UI 117 may include the same or similar features included in UI 116, such as physical and / or virtual control elements, one or more display screens, etc. In addition, each additional UI 117 may be configured to receive the same or different video and / or audio content from computing system 120 and display the content independently of the other UIs 117. For example, computing system 120 may generate a first set of visual content on a first display screen associated with a first UI 117; generate a second, different set of visual content on a second display screen associated with a second UI 117; generate a third, different set of visual content on a third display screen associated with a third UI 117; and so on. Similarly, a first occupant of the vehicle 100 may interact with the first UI 117 to generate a first set of visual content on a first display screen associated with the first UI 117; a second occupant of the vehicle 100 may interact with the second UI 117 to generate a second, different set of visual content on a second display screen associated with the second UI 117; a third occupant of the vehicle 100 may interact with the third UI 117 to generate a third, different set of visual content on a third display screen associated with the third UI 117; etc. The visual content groups may include associated audio content, which may be played at corresponding speakers 114.

[0033] Furthermore, in various embodiments, each UI 117 may additionally or alternatively be located at, in, or on a window of the vehicle 100, and each UI 117 may be configured to project visual content on the window. Specifically, the visual content may be projected to generate an AR display in conjunction with elements of the surrounding environment. For example, in Figure 1In the illustrated embodiment, the vehicle 100 includes four AR touch screen windows 115 on the right side 174 and the left side 176 of the vehicle, which can also be UI 117. In other embodiments, a greater or lesser number of AR touch screen windows 115 can be included in the vehicle 100. In some embodiments, all side windows of the vehicle 100 can be AR touch screen windows. In other embodiments, a portion of the side windows of the vehicle 100 can be AR touch screen windows. Each AR touch screen window 115 can include a touch screen portion on which objects and information can be displayed, where a user can interact with the objects and information by touching the portion of the touch screen corresponding to the objects and information.

[0034] Each AR touchscreen window 115 may be configured to display or project visual content on the corresponding AR touchscreen window 115. Specifically, as described in more detail below, each AR touchscreen window 115 may be configured to display information about the surrounding environment captured via one or more cameras 119 on the corresponding side window (and / or display screen of UI 117). The information may include, for example, text, images, and / or video related to elements of the surrounding environment.

[0035] For example, when driving through an urban environment, an occupant may select, via the touch screen of the AR touch screen window 115, a city landmark that can be seen through the corresponding AR touch screen window 115. Images of the city landmarks may be captured by one or more cameras 119 of the vehicle and displayed on a portion of the corresponding AR touch screen window 115. The computing system 120 may identify the city landmarks in real time and retrieve information about the city landmarks from the Internet. The information may be displayed to the occupant within the selected image.

[0036] Furthermore, in various embodiments, additional AR touchscreens may be configured to deliver visual content and / or audio content at one or more other AR touchscreen windows 115 and / or other UIs 117. For example, a first occupant may select a first element of the surrounding environment via the first AR touchscreen window 115, and have the first visual content displayed on one or more AR touchscreen windows 115 and / or display screens associated with other UIs 117 of other occupants of the vehicle 100. In this way, an occupant of the vehicle 100 may share information about a portion of the surrounding environment with other occupants of the vehicle 100.

[0037] For example, a first occupant of the rear seat 108 may see features of the surrounding environment, such as a mountain range, through the first AR touchscreen window 115 on the right side 174 of the vehicle 100. The first occupant may select the mountain range by touching the touchscreen portion of the first AR touchscreen window 115. Information about the mountain range may be generated by the computing system 120 and projected on the first AR touchscreen window 115 so that the first occupant may view the information while viewing the mountain range. For example, when the first occupant selects the mountain range, an image of the mountain range may be generated and "frozen" on a portion of the window, and information may be displayed near or around the mountain range in the image.

[0038] The first occupant may find information of interest and may wish to share the image and information from the first AR touchscreen window 115 with the second occupant in the rear seat 110. The first occupant may edit, annotate, or add an audio clip to the image. The first occupant may send the image and associated content to the second occupant's second AR touchscreen window or UI 117. In this way, the occupants of the vehicle may learn about elements of the surrounding environment and share what they learn with the occupants sitting in the other seats.

[0039] The computing system 120 includes a processor 142 configured to execute machine-readable instructions stored in a memory 144. The processor 142 may be single-core or multi-core, and the program executed by the processor 142 may be configured for parallel processing or distributed processing. In some embodiments, the processor 142 is a microcontroller. The processor 142 may optionally include various components distributed in two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 142 may be virtualized and executed by a remotely accessible networked computing device configured as a cloud computing configuration. For example, the computing system 120 may be communicatively coupled to a wireless network.

[0040] The computing system 120 may include a DMS 147 that can monitor the driver of the vehicle 100. For example, the camera 118 may be located in front of the vehicle 100 (e.g., on the dashboard 102) or on the side of the vehicle 100 near the driver's seat 104 and may be positioned to view the driver's face. The DMS may detect facial features of the driver. In some embodiments, the DMS 147 may be used to retrieve a driver profile of the driver based on facial features, which may be used to customize the position of the driver's seat 104, the steering wheel 112, and / or other components or software of the vehicle 100.

[0041] Computing system 120 may include OMS 148, which may monitor one or more occupants of vehicle 100. For example, camera 118 may be located on a side of vehicle 100 near one or more of passenger seats 106, 108, and 110 and may be positioned to view the faces of occupants of the vehicle.

[0042] The computing system 120 may include an ADAS 149 that can provide assistance to the driver based at least in part on the DMS 147. For example, the ADAS 149 may receive data from the DMS 147, and the ADAS 149 may process the data to provide assistance to the driver. The computing system 120 and the ADAS 149 may additionally include one or more artificial intelligence (AI) and / or machine learning (ML) / deep learning (DL) models that may be used to perform various automated tasks associated with the image data captured by the cameras 118 and 119. Specifically, the AI / ML / DL models may be used to select and generate information to be selectively displayed on a display screen for one or more occupants of the vehicle 100 and / or projected on a window, as described in more detail below. For example, a first ML model of ADAS 149 may selectively display information related to operating the vehicle 100 to a driver's display screen, or on a portion of the windshield 113 near the driver, and a second ML model of computing system 120 (e.g., an in-vehicle AR system) may selectively display educational and / or entertainment information on a display screen or window of one or more passengers of the vehicle 100 .

[0043] The computing system 120 may be configured to communicate with the user through a communication channel such as The processor 142 may be configured as a graphics processing unit (GPU), including a parallel computing architecture and parallel processing capabilities.

[0044] In addition, memory 144 may include any non-transitory tangible computer-readable medium having programmed instructions stored therein. As used herein, the term "tangible computer-readable medium" is expressly defined to include any type of computer-readable storage. The example methods described herein may be implemented using encoded instructions (e.g., computer-readable instructions) stored on a non-transitory computer-readable medium such as flash memory, read-only memory (ROM), random access memory (RAM), cache, or any other storage medium in which information is stored for any duration (e.g., for an extended period of time, permanently, for a short-lived instance, for temporary buffering, and / or for caching information).

[0045] Computer memory of computer-readable storage media as referred to herein may include volatile and nonvolatile or removable and non-removable media for storing electronically formatted information such as computer-readable program instructions or computer-readable program instruction modules, data, etc., which may be independent or part of a computing device. Examples of computer memory may include any other medium that can be used to store information in a desired electronic format and can be accessed by at least a portion of one or more processors or computing devices. In various embodiments, memory 144 may include an SD memory card, an internal and / or external hard disk, a USB memory device, or similar modular memory.

[0046] In addition, in some examples, the computing system 120 may include multiple subsystems or modules that perform specific functions related to performing image acquisition and analysis tasks. As used herein, the terms "system," "unit," or "module" may include hardware and / or software systems that operate to perform one or more functions. For example, a module, unit, or system may include a computer processor, controller, or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer-readable storage medium such as a computer memory. Alternatively, a module, unit, or system may include a hard-wired device that performs operations based on the hard-wired logic of the device. The various modules or units shown in the figures may represent hardware that operates based on software or hard-wired instructions, software that directs hardware to perform operations, or a combination thereof.

[0047] Reference now Figure 2 , block diagram 200 shows an example of a vehicle computing system 202 of a vehicle, which may be the vehicle computing system 202 of the vehicle ... Figure 1 1 is a non-limiting example of a computing system 120 depicted in FIG. 100 of a vehicle. The vehicle computing system 202 can be communicatively coupled to one or more internal cameras 218 of the vehicle (e.g., one or more cameras 118) and one or more external cameras 219 of the vehicle (e.g., one or more cameras 119). The internal cameras 218 and the external cameras 219 can provide image data to the vehicle computing system 202.

[0048] The computing system 202 includes a processor 204 configured to execute machine-readable instructions stored in a non-transitory memory 206. The processor 204 can be single-core or multi-core, and the programs executed thereon can be configured for parallel processing or distributed processing. In some embodiments, the processor 204 can optionally include various components distributed in two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 204 can be virtualized and executed by a remotely accessible networked computing device configured as a cloud computing configuration.

[0049] The non-transitory memory 206 includes an onboard AR system 208 that can manage the generation of visual and audio content for the occupants of the vehicle, as described above with reference to Figure 1 Specifically, the vehicle-mounted AR system 208 may include an AR module 210 that can generate and project visual content on one or more windows of the vehicle for the occupants to view in conjunction with elements of the vehicle's surrounding environment. The vehicle-mounted AR system 208 may include instructions that, when executed by the processor 204, cause the vehicle computing system 202 to perform one or more steps of the method 400, which will be referred to below. Figure 4 Describe in more detail.

[0050] The content management module may additionally store an AI module 211, which may include one or more AI models and instructions for implementing the one or more AI models, as described in more detail below. The one or more AI models may be ML or DL ​​neural network models. The AI ​​module 211 may include trained and / or untrained neural networks, and may also include various data or metadata related to the one or more neural networks stored therein. The AI ​​module 211 may also include other types of statistical models, rule-based models, or other models. The AI ​​module 211 may include instructions for the following operations: implementing one or more gradient descent algorithms, applying one or more loss functions and / or training routines for adjusting parameters of one or more neural networks of the AI ​​module 211. The AI ​​module 211 may include a training data set for one or more AI models of the AI ​​module 211. The AI ​​module 211 may include instructions for deploying one or more trained AI models. In addition, the AI ​​module 211 may include instructions that, when executed by the processor 204, cause the vehicle computing system 202 to perform one or more steps of the method 300, which will be referred to below. Figure 3 Describe in more detail.

[0051] It will be appreciated that in some examples, the vehicle-mounted AR system 208 may additionally or alternatively be accessed via a user-provided mobile computing system (such as a smartphone, computer, or tablet). The vehicle-mounted AR system 208 may additionally or alternatively be included in a networked computing system such as a cloud computer or similar computing environment and may be remotely accessed (e.g., via a wireless network such as a Figure 1 As such, although Figure 2 208 is shown within the vehicle computing system 202 , but in other embodiments, at least a portion of the vehicle AR system 208 may be stored in a computing device and / or network external to the vehicle computing system 202 that is communicatively coupled to the vehicle computing system 202 .

[0052] The vehicle computing system 202 can be coupled to various UI elements 216, which can include one or more speakers 214, one or more display devices 220, one or more projectors 222, one or more head mounted displays 224, and one or more AR touch screen windows 232. The UI elements 216 can include Figure 1 1 and 12. Non-limiting examples of AR touch screen window 115, UI 116, and / or UI 117 of vehicle 100 of the present invention. As such, one or more speakers 214 may be the same or similar to speaker 114, and AR touch screen window 232 may be the same or similar to AR touch screen window 115.

[0053] One or more display devices 220 and one or more projectors 222 may be used to display visual content to occupants of the vehicle. The one or more display devices 220 may include one or more display screens utilizing virtually any type of technology. The one or more projectors 222 may be configured to project the visual content onto one or more windows of the vehicle. As an alternative to the one or more display devices and / or one or more projectors, one or more head mounted displays may optionally be used by one or more occupants to view the visual content. Although not described herein, the one or more display devices 220 may include one or more display screens utilizing virtually any type of technology. The one or more projectors 222 may be configured to project the visual content onto one or more windows of the vehicle. Figure 2 , but UI elements 216 may also include one or more user input devices, which may include one or more of a touch screen, a keyboard, a trackpad, a microphone, a motion sensing camera, or other devices configured to enable a user to interact with and manipulate data within the vehicle computing system 202.

[0054] The non-transitory memory 206 also includes an ADAS module 212, which provides assistance to the driver of the vehicle, and a DMS module 240 and an OMS module 242, which monitor the vehicle operator and the occupant, respectively. ADAS 212 and / or DMS 240 can detect the specific state, attributes and posture of the driver, and can use one or more UI elements 216 to notify the occupant. As an example, the ADAS module 212 can be connected to one or more UI elements 216 to output an alarm via a display device 220 and / or a speaker 214.

[0055] The vehicle computing system 202 may include an onboard navigation system 244. The onboard navigation system 244 may provide route information to the driver, including the current location of the vehicle and the destination of the vehicle. For example, when operating the vehicle, the driver may input the destination into the onboard navigation system 244. The onboard navigation system 244 may indicate one or more routes from the current location to the destination on a map displayed by the onboard navigation system 244. In various embodiments, the map may be displayed on a screen of a display (e.g., a dashboard display) of the onboard navigation system 244. The driver may select a route in one or more routes, and the onboard navigation system 244 may provide instructions to the driver and / or indicate the vehicle's progress toward the destination on a map. The onboard navigation system 244 may additionally calculate information, such as an estimated distance to the destination, an estimated time to reach the destination based on the speed of the vehicle, traffic instructions on the route, and / or other usage information to the driver.

[0056] The onboard navigation system 244 may rely on a global positioning system (GPS) to determine the current location of the vehicle. In various embodiments, the current location, route, and / or other GPS-based information may be used by one or more ML models of the AR module 210 to identify elements of the vehicle's surroundings.

[0057] The vehicle computing system 202 may include a communication module 250 that can manage wireless communications between the vehicle computing system 202 and one or more wireless networks 260. In various embodiments, the wireless network 260 may include the Internet. The wireless communications managed by the communication module 250 may include, for example, retrieving information available on the Internet about image elements captured by the external camera 119. In addition, the wireless network 260 may include one or more private networks. For example, the communication module 250 may be used to send images collected by the external camera 119 to a cloud-based database 262 via a private network for processing. In some embodiments, the images may be used as training data to train one or more ML models of the AI ​​module 211, for example, to classify elements in the image as high priority elements or low priority elements, and / or to train one or more ML models of the AR module 210, for example, to identify elements in the image, retrieve data about the identified elements, and / or selectively display the retrieved data to passengers of the vehicle, as described in more detail below.

[0058] It should be understood that Figure 2 The vehicle computing system 202 shown is for illustration and not limitation. Another suitable vehicle computing system may include more, fewer, or different components.

[0059] Reference now Figure 3 , showing a method for processing images captured by one or more cameras (e.g. Figure 1 Method 300 for processing image data captured by an external camera 119 of a vehicle 100. Specifically, method 300 can be used to determine whether an element (e.g., an object) of the vehicle's surrounding environment is a high priority element or a low priority element. Figure 2 The computing system 120 and / or the vehicle computing system 202) may be configured to determine the associated actions to be taken by the computing system 120 and / or the vehicle computing system 202 while the vehicle is moving. For purposes of this disclosure, high priority elements may include visual elements of the surrounding environment that may be relevant to the operation of the vehicle by the vehicle operator or the safety of the vehicle. In contrast, low priority elements may include visual elements of the surrounding environment that are not relevant to the operation of the vehicle or the safety of the vehicle. For example, low priority elements may include natural or man-made landmarks of the surrounding environment that may be of interest to passengers of the vehicle. The method 300 may be executed by a processor of the computing system (such as Figure 2 The processor 204) executes based on instructions stored in an AI module (such as AI module 211) in the non-volatile memory of the computing system.

[0060] The method 300 begins at 302, where the method 300 includes receiving a selection of an element of the surrounding environment of the vehicle. In various embodiments, as described above, the element can be selected by an occupant of the vehicle (e.g., a passenger) via the AR touch screen window. For example, the occupant can touch a location on the AR touch screen window where the element is visible through the AR touch screen window, and the computing system can map the location to the element of the surrounding environment. In one embodiment, the computing system can select the element based on the occupant's use of one or more seat sensors (such as, Figure 1 The computing system can estimate the position of the occupant's eyes based on the weight and / or size determined by the passenger seat sensors 126 and 128 (e.g., weight sensors, seat position / height sensors, etc.) of the vehicle 100. The computing system can draw a track between the estimated position of the occupant's eyes and the position on the AR touch screen window, and can determine the selected element based on the track. For example, the computing system can use triangulation to determine the intersection between the track and the camera angle of the vehicle camera. In other examples, different methods can be used to determine the selected element.

[0061] At 304, method 300 includes: capturing an image of the surrounding environment of the vehicle including the selected element when the occupant selects the element, and detecting the selected element in the image. The image can be captured by an external camera of the vehicle (e.g., camera 119). In some examples, the image of the element can be captured by two or more different external cameras of the vehicle. For example, a first image of the element can be captured by a first external camera installed in the side-view mirror of the vehicle, and a second image of the same element can be captured by a front-end camera of the vehicle. In such examples, the largest and / or most detailed image of the first image and the second image can be selected. In other examples, the image captured by the camera closest to the occupant can be selected. In various embodiments, the selected element can be referenced by a position pointer, a bounding box, or different graphical elements indicating the position of the selected element within the image.

[0062] The selected element may be detected in the image using a first ML model. The first ML model and other ML models described herein may be stored in an AI module (e.g., Figure 2 In various embodiments, the first ML model may include a convolutional neural network (CNN) comprising multiple layers, wherein low-level features of the image are detected at one or more of the multiple layers, and elements of the image are identified based on the detected low-level features according to various techniques and methods known in the art. For example, the CNN may identify a building, or a tree, or a bridge, etc. in the image. In addition, in some embodiments, the computing system may display a prompt (e.g., a dialog box, etc.) on the AR touch screen window, requesting the occupant to disambiguate between two or more elements located in the same area of ​​the surrounding environment from the occupant's perspective.

[0063] At 306, method 300 includes identifying a selected element in the image. In some embodiments, the selected element can be identified by performing one or more Internet image searches on the selected element (e.g., after performing segmentation of the image). GPS data can also be used to determine the selected element. For example, GPS data can be used to identify candidates for the selected element, and images of the candidates can be retrieved from one or more local and / or online databases, online reference sites, or Internet websites and compared to the image of the selected element.

[0064] At 308, method 300 includes: classifying the detected elements based on priority. In some embodiments, a second ML model different from the first ML model may be used to classify the detected elements. The second ML model may classify the detected elements into one of two categories: a high priority category and a low priority category. The high priority category may include elements that represent potential dangers to the vehicle and / or elements that can be used by the driver to make decisions when driving the vehicle. For example, elements of the high priority category may include one or more vehicles present in the image, such as vehicles operating in the same or adjacent lanes of vehicles in front of, behind, or beside the vehicle; obstacles detected on the road on which the vehicle is traveling; traffic lights or warning lights in the image, such as railroad crossings; and the like. Elements of the low priority category may include objects in the surrounding environment that are not related to the operation of the vehicle. Elements of the low priority category may include architectural elements of interest, such as buildings or bridges; monuments, statues, or other man-made artistic expressions; natural and / or geographical features, such as trees, mountains, rivers, oceans, lakes, and the like; and the like. The second ML model may provide a confidence score that the detected element is in the high priority category.

[0065] In various embodiments, the second ML model can be a classification neural network (NN) trained on images including various examples of high priority elements and low priority elements. In some embodiments, examples can be collected from external cameras of multiple vehicles when the vehicles are operating in different environments, and these examples can be wirelessly transmitted to a cloud-based storage device to be incorporated into one or more training, testing and / or validation data sets. These training sets can then be used to train the second ML model. After training, the second ML model can receive images including detected elements as input and output a classification of the detected elements as high priority elements or low priority elements.

[0066] At 310, method 300 includes determining, for each detected element in the image, whether the detected element is a high priority element. For example, if the confidence score of the detected element being a high priority element output by the second ML model is higher than a threshold confidence score, the detected element can be determined to be a high priority element.

[0067] If the detected element at 310 is determined to be a high priority element, the method 300 proceeds to 312. At 312, the method 300 includes: defaulting the ADAS system of the vehicle to process the detected element. In other words, the ADAS system can be configured to warn or notify the driver about the high priority element captured by one or more cameras, and the computing system does not take a separate action for the high priority element.

[0068] Alternatively, if the element detected at 310 is determined to be a low priority element (e.g., a non-high priority element), method 300 proceeds to 314. At 314, 300 includes processing the low priority element for passenger education and / or entertainment. Processing the low priority element for passenger education and / or entertainment may include generating information related to the low priority element and displaying it to one or more passengers of the vehicle. Figure 4 Describes the handling of low priority elements in more detail.

[0069] Reference now Figure 4 , showing a method for processing a video image captured by one or more cameras (e.g. Figure 1 Method 400 for detecting low priority elements in image data captured by an external camera 119 of a vehicle 100. In various embodiments, the low priority elements may be detected by a computing system of the vehicle (such as Figure 2 The computing system 202 of the vehicle is detected according to the above method 300. The low priority cells can be processed to generate filtered relevant information of the low priority cells and displayed to one or more passengers of the vehicle for educational and / or entertainment purposes. The method 400 can be performed by a processor of the computing system (such as Figure 2 The processor 204) executes based on instructions stored in an AR module (such as AR module 210) in the non-volatile memory of the computing system.

[0070] Method 400 begins at 402, where method 400 includes: from an AI module (e.g., Figure 2The AI ​​module 211 of the vehicle receives an image of a selected low priority element within the surrounding environment of the vehicle. As described above, one or more ML models of the AI ​​module may have detected the low priority element in the image. In response to detecting the low priority element, the image can be transmitted to one or more ML models of the AR module, which can process the image for educational and / or entertainment purposes.

[0071] At 404, method 400 includes identifying and obtaining information about a detected low priority element included in the image. Identifying the low priority element may include associating the low priority element with a wireless network (e.g., Figure 2 Low priority elements may be identified based in part on the current location of the vehicle based on GPS data retrieved from an onboard navigation system of the vehicle (e.g., onboard navigation system 244). For example, the GPS data may be used to reference a first online database of buildings corresponding to an address at the current location of the vehicle to determine a match with a building in the surrounding environment. Alternatively, the GPS data may be used to reference a second online database of geographic features near the current location.

[0072] Additionally or alternatively, identifying low priority elements may include segmenting the element in the image and matching the segmented element to similar images in one or more local and / or online databases, reference sites, and / or websites on the Internet. For example, a landmark building may be matched to images of landmark buildings on a website (e.g., a travel or architectural website), or a well-known mountain range may be matched to images of mountains found on a website. A template matching search may be performed on the relevant database.

[0073] Once the element is identified, one or more identifiers for the element may be retrieved from local and / or online databases and / or reference sites, and the one or more identifiers may be used to perform one or more searches for information about the element. In various embodiments, obtaining information about detected low-priority elements included in an image may include sequentially performing one or more online searches. A first search may be performed to retrieve a first set of information; elements of the first set of information may be used to perform a second search to retrieve a second set of information; elements of the second set of information may be used to perform a third search to retrieve a third set of information; and so on.

[0074] The amount of information to be retrieved can be determined by one or more AI models of the AR module. In some embodiments, the amount of information to be retrieved can be determined by a rule-based model based on the type of visual element and / or other criteria. For example, if a building is identified as a museum, the information retrieved about the museum may include the museum's current or permanent exhibitions, the museum's opening hours, and the cost of visiting the museum. If the building is identified as a historical building or a landmark building, the information retrieved about the building may include the architect of the building, a brief summary of the museum's history, and the address of the building.

[0075] In some embodiments, obtaining information about the low priority element may include searching for and downloading images of the low priority element captured from other camera angles. For example, the low priority element may be a mountain range, where images of the mountain range taken from various sides, fronts, and / or perspectives of the mountain range may be found on the Internet. The images may be downloaded so that if the passenger wishes to view the mountain range from different sides, the passenger may use the touch screen of the AR touch screen window to rotate the mountain range and view one or more images of the mountain range taken from different sides, fronts, and / or perspectives, as described below with reference to FIG. Figure 7 described.

[0076] At 406, method 400 includes: filtering and editing the obtained information. The obtained information may include an amount of text that is too large to be effectively displayed next to the image of the element on the AR touch screen window, and the amount of text can be reduced to a desired amount. For example, the element can be a museum that the vehicle is passing by. One or more AI models and / or rule-based models can obtain the name and description of the museum from an online travel website. One or more AI models and / or rule-based models can use the name to search the museum's website. One or more AI models and / or rule-based models can retrieve the museum's opening hours from the homepage of the museum's website. One or more AI models and / or rule-based models can retrieve a paragraph describing the museum's current exhibition from the second page of the museum's website.

[0077] The name, description, opening hours, and a paragraph describing the museum's current exhibit may include too much text to be clearly displayed near the cropped image of the selected element on the AR touchscreen window. As a result, one or more AI models and / or rule-based models may reformat the text to include a list of items, where the list of items includes a minimum number of words. Additionally, the paragraph may be edited to shorten the length of the content of the paragraph to one or two sentences, or in some cases to a subtitle.

[0078] At 408, the method includes: displaying the image and the edited information on the AR touch screen window. Displaying the image and the edited information on the AR touch screen window may include: generating a cropped image of the selected element, and scaling the cropped image to display on the AR touch screen window. In various embodiments, a portion of the AR touch screen window may be allocated for displaying the selected element. For example, the portion of the AR touch screen window may include an area of ​​the AR touch screen window located in the lower right corner of the AR touch screen window, which corresponds to approximately 10% of the total area of ​​the AR touch screen window. It should be understood that the examples provided here are for illustrative purposes, and in other embodiments, the portion may include larger or smaller sized areas of the AR touch screen window that may be located at different locations within the AR touch screen window.

[0079] Brief reference Figure 5 , showing a vehicle (such as Figure 1 The side window 500 is an example of an AR touch screen window, as described above with reference to FIG. Figure 4 Through the side window 500, a passenger sitting in the passenger seat of the vehicle can see the surrounding environment 504. For example, the passenger seat can be the rear right passenger seat of the vehicle. The vehicle is driven by the driver in the direction indicated by the arrow 520, so that elements of the surrounding environment 504 can enter the field of view on the first side 510 of the side window 500 and disappear from the field of view on the second side 512 of the side window 500. In the surrounding environment 504, the passenger can view the element of interest 502. Figure 5 , element 502 may be a historic church. When the church comes into view, the passenger may be interested in learning more about the church. To learn more about the church, the passenger may touch the side window 500 at point 506 to select element 502, as shown by hand icon 508. When the passenger touches the side window 500, the touch may be detected by the touch screen of the side window 500, which may indicate to the vehicle's computing system (e.g., computing system 120 and / or vehicle computing system 202) that the passenger has selected element 502. The computing system may identify the church as element 502, for example, by comparing the passenger's estimated line of sight from the position of the passenger's eyes through point 506 on the side window 500 to the camera angle of an external camera of the vehicle (e.g., camera 119). In some embodiments, a GPS system of an onboard navigation system (e.g., onboard navigation system 244) may be used to help identify the selected element 502. A cropped and expanded image of element 502 may then be displayed in the side window 500, as shown in FIG. Figure 6 shown.

[0080] Figure 6 504 in response to a passenger selection element 502 in the surrounding environment 504. Figure 5The cropped and expanded image 600 of the element 502 is displayed in the side window 500 of FIG. When the cropped and expanded image 600 is displayed on the side window 500, Figure 5 The surroundings 504 including the church may no longer be visible and may be replaced by surroundings 604 not including the church. In other words, element 502 may disappear from view via side window 500, and surroundings 604 may include different elements. For example, surroundings 604 does not include element 502, but includes element 620 (e.g., a house).

[0081] The cropped and expanded image 600 is depicted as having a width 607 and a height 606. In the depicted embodiment, the width 607 is approximately 1 / 6 of the total width 608 of the side window 500, while the height 606 is just under half of the total height 610 of the side window 500. Thus, the cropped and expanded image 600 occupies a relatively small portion of the side window 500, allowing a large portion of the surrounding environment 604 to be viewed through the side window 500 without being obscured by the cropped and expanded image 600. Additionally, although not shown in FIG. Figure 6 , but in some embodiments, the cropped and expanded image 600 is partially transparent, and some elements of the surrounding environment 604 can be viewed behind the cropped and expanded image 600. In other embodiments, the width 607, height 606, and corresponding dimensions of the cropped and expanded image 600 are compared Figure 6 100 may be larger or smaller than that depicted and / or may occupy a greater or lesser percentage of the total area of ​​the side window 500 .

[0082] In addition, as described above, information such as text information may be superimposed on the cropped and expanded image 600. Figure 6 In the embodiment of the present invention, subtitle 630 indicates that the object included in the image 600 through cutting and expansion is a historic church. In other embodiments, different and / or additional information can be superimposed and displayed on the image 600 through cutting and expansion. Different and / or additional information can be displayed as, for example, a list of items or a list of sub-items in a paragraph, in a series of sentences or in a different way. For example, different and / or additional information can include historical facts or other information of the church. This information can include the top, bottom, side of the church or in the middle of the church. In some embodiments, if the size of different and / or additional information is greater than the image 600 through cutting and expansion, different and / or additional information can be displayed in a scrollable view, wherein the user can use a finger to scroll up or down to view the different parts of different and / or additional information.

[0083] In various embodiments, the intensity or other properties or attributes / characteristics of the image may be adjusted to make it easier to read the information displayed on the cropped and expanded image 600. Additionally, color may be used to increase the contrast between the information and the cropped and expanded image 600. For example, the information may be displayed in one or more colors and / or the color attributes of the cropped and expanded image 600 may be adjusted.

[0084] Back to Figure 4 At 410, method 400 includes determining whether a user interaction with the AR touch screen window has been detected. After the cropped and expanded image is displayed on the AR touch screen window, the passenger can interact with the cropped and expanded image in various ways, such as with a finger (e.g., swiping, selecting, double-clicking, etc.). If a user interaction is detected at 410, method 400 proceeds to 412. At 412, the method includes: responding to the user interaction accordingly with various predetermined actions, as described below with reference to Figure 7 described.

[0085] Alternatively, if no user interaction is detected at 410, the method proceeds to 414. At 414, the method includes continuing to display the cropped and expanded image in the AR touch screen window until a user interaction is detected. For example, the passenger may perform a user interaction to remove (e.g., stop displaying) the cropped and expanded image. The method 400 ends.

[0086] Figure 7 A plurality of interactions 700 that can be performed on the cropped and extended image 600 of element 502 are shown, as depicted by various hand icons. Each interaction can be associated with different predetermined actions or operations of the computing system. The passenger can swipe right on the cropped and extended image 600, as indicated by hand icon 702, or swipe left on the cropped and extended image 600, as indicated by hand icon 706. The passenger can tap or double-click the cropped and extended image 600, as indicated by hand icon 708. The passenger can tap the cropped and extended image 600 with two fingers, as indicated by hand icon 704. The passenger can rotate the cropped and extended image 600 to view the image of element 502 from other viewpoints (e.g., by sliding a finger on the cropped and extended image 600 to achieve a 360-degree view of element 502, as indicated by hand icon 710. Images of element 502 from other viewpoints can be downloaded from the Internet.

[0087] In other embodiments, more or less user interactions may be supported. In addition, it should be understood that the examples provided herein are for illustrative purposes, and different functions may be provided using the above interactions or different interactions without departing from the scope of the present disclosure.

[0088] Figure 7 Each interaction 700 depicted in the figure can be mapped to a different operation or action performed by the computing system of the vehicle. For example, tapping the cropped and expanded image 600 (or touching and holding a predetermined length of time) can open a menu or wizard that allows passengers to select from multiple options, such as saving, deleting and / or sharing images, annotating images, obtaining more information about images, etc. Other user interactions can provide shortcuts to menus or wizards. For example, in one embodiment, swiping left on the cropped and expanded image 600 can remove (e.g., remove) the cropped and expanded image 600 from the side window 500, so that the cropped and expanded image 600 is no longer displayed in or on the side window 500. Double-clicking the cropped and expanded image 600 can open an Internet browser, and its search page is pre-filled with search result information about element 502. For example, the name of element 502 determined during the identification of element 502 described in reference method 400 above can be used to perform an Internet search. The Internet browser can be superimposed on the cropped and expanded image 600, or displayed on different parts of the side window 500. The user may resize the internet browser, for example, by dragging a corner of the internet browser.

[0089] As another example, tapping the cropped and expanded image 600 with two fingers can display a virtual keypad from which the passenger can type information, add images from an album stored in a smart phone or vehicle, record audio clips, or associate other media with the cropped and expanded image 600. For example, element 502 (e.g., a church) can remind the passenger of different churches encountered during the journey. The passenger can tap the cropped and expanded image 600 with two fingers to pop up a virtual keypad. The passenger can use the virtual keyboard to type a short note referring to different churches, or use an Internet browser to search for different photos of the church or different churches. When the passenger types information or adds different photos, this is marked to element 502 (e.g., a point of interest (POI) object). After adding, the passenger can view the added information and / or different photos in the cropped and expanded image 600. All information (whether obtained by the system from the Internet or added by the passenger) can be visible in the cropped and expanded image 600 and / or can be viewed by scrolling.

[0090] As another example, swiping right on the cropped and expanded image 600 may pop up a dialog box 720 on the side window 500 that may allow the passenger to request to share the cropped and expanded image 600 with one or more other passengers of the vehicle. For example, the dialog box 720 may prompt the passenger to select one or more other passengers of the vehicle from a list with whom the cropped and expanded image will be shared. Figure 7As shown, one or more other passengers can be referenced by seat position in the dialog box. For example, for a vehicle with four seats, options for the front left seat passenger, the rear left seat passenger, and the rear right seat passenger can be provided in the dialog box 720. The seat options of the passengers who share information can be deselected. When the passenger selects one or more passengers and selects the submit button of the dialog box, one or more corresponding requests can be sent to one or more passengers, indicating that the passenger wishes to share the image. These requests can be displayed in the corresponding dialog box displayed on the corresponding AR touch screen window of one or more passengers. If a passenger in one or more passengers accepts the request, the cropped and extended image can be displayed on the AR touch screen window of the accepting passenger. The cropped and extended image can continue to be displayed on the AR touch screen window of the sending passenger until it is removed by the sending passenger.

[0091] Alternatively, in some embodiments, the passenger can share the cropped and expanded image 600 and associated mixed media content with friends or family via email using the vehicle's wireless network. For example, dialog box 720 may include an option to create an email and type an email address using a virtual keyboard. The computing system may communicate with the vehicle via a communication module (e.g., Figure 2 The communication module 250 and the wireless network 260) send the cropped and expanded image 600 and the associated mixed media content to an email address.

[0092] In various embodiments, the passenger can interact with the AR touch screen window to create a travel log that records the places visited by the passenger during the journey, including images of elements of interest encountered in the surrounding environment along the journey. For example, the travel log can be a map-based representation of the journey, where the route driven by the vehicle is stored by the computing system, and links to (e.g., marked) images are shown on the map. The passenger can select one of these links, and one or more annotated images corresponding to the location on the route where the image was taken can be displayed. In some examples, the passenger can select "Start Travel Log" from the menu to create a new travel log, and select "End Travel Log" to stop saving images to the travel log. The computing system can store the map-based log in a cloud-based storage device (e.g., a cloud-based database 262), and the passenger can download the map-based log for viewing on a smart phone, tablet computer, personal computer, etc. In addition or alternatively, the passenger can create a travel planner for future travel, where images and text are associated with places that the passenger may wish to visit in the future.

[0093] As described above, images can be annotated with text such as notes, thoughts, reflections, memories, etc. In addition, other types of media can be associated with images, such as sound files (e.g., voice narration, music, etc.), video files (e.g., video clips captured with a smartphone or downloaded from the Internet via a smartphone or vehicle's computing system), or other types of files. For example, an in-vehicle AR system (e.g., Figure 2 The in-vehicle AR system 208 can be configured to receive voice commands, where the various functions described above can be performed via predefined commands (e.g., “add a note,” “open a browser,” etc.).

[0094] Thus, systems and methods are provided that allow passengers of a vehicle to interact with the surrounding environment in an entertaining and educational manner, which can alleviate boredom, particularly during long journeys or severe congestion. Elements of the surrounding environment can be explored in a comfortable and self-directed manner, where customized information can be obtained from the Internet and / or other online or local sources based on the specific interests of the passenger. Passengers can capture images of elements of the surrounding environment, annotate the images with text, sound, video, and / or other images to create mixed media content, and share the mixed media content with other passengers or friends over the Internet. For example, schoolchildren on a study tour can participate in a fun way, can share the knowledge gained during the journey, and can access the knowledge later. Passengers can plan future journeys and save the plans in a customized trip planner in the cloud to follow later. Passengers of all ages can record their journeys in a multimedia travel log or album, saving images and information collected along the journey for later viewing and / or sharing.

[0095] The technical effect of allowing passengers to select and capture images of their surroundings, and annotate, save and share the selected images, is that passengers can access real-time information about the area they are traveling in that is responsive to their inputs in the augmented reality environment.

[0096] The present disclosure also provides support for a method for a computing system for a vehicle, the method comprising: in response to a passenger of the vehicle selecting an element of the surrounding environment of the vehicle that can be seen by the passenger through a window of the vehicle: capturing a first image of the surrounding environment including the element via one or more cameras of the vehicle; detecting and identifying an element of the first image; retrieving information about the element from the Internet; and displaying a second image of the element and the retrieved information about the element on the window. In a first example of the method, the one or more cameras of the vehicle include at least a front camera, a rear camera, a first side camera mounted on the right side of the vehicle, and a second side camera mounted on the left side of the vehicle. In a second example of the method, optionally including the first example, the element is selected by the passenger via a touch screen portion of the AR touch screen window. In a third example of the method, optionally including one or both of the first and second examples, detecting the element of the first image also includes: comparing the estimated line of sight of the passenger from the position of the passenger's eyes through the selected position of the AR touch screen window with the camera angle of a camera in the one or more cameras of the vehicle. In a fourth example of the method, optionally including one or more or each of the first to third examples, identifying the element of the first image further includes: determining the location of the element within the surrounding environment using a global positioning system (GPS) of the vehicle. In a fifth example of the method, optionally including one or more or each of the first to fourth examples, identifying the element of the first image further includes: segmenting the element within the first image, and matching the segmented element with one or more images of the element retrieved from one or more local and / or online databases and online reference sites or Internet websites. In a sixth example of the method, optionally including one or more or each of the first to fifth examples, retrieving information about the element from the Internet further includes: retrieving one or more identifiers of the element from one or more local and / or online databases, reference sites and Internet websites, and performing one or more searches for information about the element in one or more local and / or online databases, reference sites and Internet websites using the one or more identifiers. In a seventh example of the method, optionally including one or more or each of the first to sixth examples, the method further includes: performing a first search to retrieve a first set of information, and performing a second search based on the elements of the first set of information to retrieve a second set of information. In an eighth example of the method, optionally including one or more or each of the first to seventh examples, displaying a second image of the element and the retrieved information about the element on the window further includes: displaying the second image in a portion of the AR touch screen window.In a ninth example of the method, optionally including one or more or each of the first to eighth examples, the method further includes: performing an operation on the second image in response to a passenger performing a user interaction with the second image via the AR touch screen window, the user interaction being one of: double-clicking the second image, swiping the second image in a first direction, swiping the second image in a second direction, touching the second image with two fingers, and sliding the passenger's finger on the second image. In a tenth example of the method, optionally including one or more or each of the first to ninth examples, the method further includes: displaying an Internet browser window with information about the element on the AR touch screen window in response to the user interaction. In an eleventh example of the method, optionally including one or more or each of the first to tenth examples, the method further includes: removing the image in response to the user interaction. In a twelfth example of the method, optionally including one or more or each of the first to eleventh examples, the method further includes: displaying a graphical element in response to the user interaction, thereby allowing the passenger to send the image to one or more other passengers of the vehicle. In a thirteenth example of the method, optionally including one or more or each of the first to twelfth examples, the method further includes: displaying one or more graphic elements on the AR touch screen window in response to the user interaction, thereby allowing the passenger to input information and / or images stored on the smart phone and / or stored in the computing system.

[0097] The system may also consider vehicle speed relative to the environment and objects being detected and identified, particularly with respect to augmented reality experiences. Because the vehicle may be operated at a wide range of speeds, the occupant's focus in the field of view may vary depending on the distance outward from the vehicle toward the horizon (e.g., the horizon outward from the vehicle's side windows). Thus, the system may utilize vehicle speed as an additional input for determining the type, quantity, and location of objects, identifying objects, and / or providing information based thereon.

[0098] In an example where identifying the element of the first image also includes comparing the estimated line of sight of the passenger from the position of the passenger's eyes through the selected position of the AR touchscreen window to the camera angle of a camera in one or more cameras of the vehicle, the identification can take into account the vehicle speed if the object is stationary. For example, at higher vehicle speeds, the line of sight can be adjusted to be further toward the horizon, for example, by using a lookup table with an adjustment factor based on vehicle speed, or vice versa. As another example, where identifying the element of the first image also includes: using the vehicle's global positioning system (GPS) to determine the position of the element in the surrounding environment, the identification can also be based on vehicle speed. For example, at higher vehicle speeds, the position can be determined to be further toward the horizon, again using an adjustment factor in a lookup table based on vehicle speed, or vice versa. The lookup table can be calibrated based on vehicle testing.

[0099] In yet another example of retrieving information about an element from the Internet, the retrieval may also be based on vehicle speed. For example, the amount of information reduced may decrease as vehicle speed increases. Thus, at higher vehicle speeds, the information may be compressed. As another example, for higher vehicle speeds, the retrieved information may be aggregated to a greater extent. The amount aggregated may be proportional to vehicle speed.

[0100] In one example, the method includes: performing a first search to retrieve a first set of information; and performing a second search based on elements of the first set of information to retrieve a second set of information, and one or both of the first search and the second search can also be based on vehicle speed. For example, the type of information can be adjusted for different vehicle speeds. At low vehicle speeds, not only can more information be retrieved (and displayed), but the information at low speeds can also be adjusted to focus on facts and graphics, while picture information is received and displayed at high vehicle speeds. Therefore, the first search and the second search can be based on vehicle speed and adjusted to search different aspects (for one or both of the first search and the second search) according to the vehicle speed.

[0101] In an example in which displaying a second image of an element on a window and the retrieved information about the element also includes displaying the second image in a portion of the AR touchscreen window, as described above, the display can be based on vehicle speed. At higher vehicle speeds, the second image can be displayed as being further away from the vehicle because occupants can naturally look further away from the vehicle as speed increases. Adjustments based on the degree of distance can be proportional to vehicle speed. As another example, as described above, the type of image provided can change with vehicle speed, with text-based images provided at lower speeds and picture images provided at higher speeds.

[0102] The present disclosure also provides support for a computing system for a vehicle, the computing system comprising: an in-vehicle AR system, a processor, and a non-transitory memory storing instructions, which instructions, when executed, cause the processor to: in response to a passenger of the vehicle selecting an element of the vehicle's surrounding environment in an AR touch screen window of the vehicle coupled to the computing system, the element being visible through the AR touch screen window: capturing a first image of the surrounding environment including the element via one or more cameras of the vehicle, detecting and identifying elements of the first image, retrieving information about the element from the Internet, and displaying a second image of the element and the retrieved information about the element on a portion of the AR touch screen window. In a first example of the system, additional instructions are stored in the non-transitory memory, which instructions, when executed, cause the processor to: perform an operation on the second image in response to the passenger performing a user interaction with the second image via the AR touch screen window, the user interaction being one of: double-clicking the second image, swiping the second image in a first direction, swiping the second image in a second direction, touching the second image with two fingers, and sliding the passenger's finger on the second image. In a second example of the system, optionally including the first example, the operation is one of: displaying an internet browser window with information about the element on the AR touch screen window, displaying a dialog box on the AR touch screen window, displaying a virtual keyboard on the AR touch screen window, rotating the second image to achieve a 360-degree view of the element, annotating the second image with text, image, audio, video, or different types of content selected by the passenger via one of a microphone, a dialog box, and a virtual keyboard, sharing the second image or the annotated second image with different passengers of the vehicle, storing the second image or the annotated second image in a passenger's travel log or travel planner stored in a cloud-based storage device, and removing the second image or the annotated second image from the AR touch screen window. In a third example of the system, optionally including one or both of the first and second examples, the travel log or travel planner includes a map-based interface, wherein a link to the second image or the annotated second image is included on the route traveled by the vehicle, and the second image or the annotated second image is displayed when the user selects the link.

[0103] The present disclosure also provides support for a method for a computing system of a vehicle, the method comprising: while the vehicle is moving on a route: in response to a passenger of the vehicle selecting an element of the surrounding environment of the vehicle that can be seen through the AR touch screen window of the vehicle via a touch screen of the AR touch screen window located next to the passenger, displaying an image of the element, retrieving information about the element from the Internet, and displaying the information superimposed on the image, and in response to the passenger interacting with the image via the touch screen, using the passenger's finger to perform one of the following: displaying an Internet browser window with search results about the element on the AR touch screen window, displaying text, images, audio, video, or different types of content entered by the passenger on the AR touch screen window, sharing the image with a different passenger of the vehicle, the different passenger selected by the passenger, rotating the image to achieve a 360-degree view of the element, and removing the image from the AR touch screen window. In a first example of the method, the method also includes: storing the image in an interactive map-based representation of the passenger's journey.

[0104] When introducing the elements of various embodiments of the present disclosure, the articles "one", "an" and "the" are intended to indicate the presence of one or more elements. The terms "first", "second", etc. do not represent any order, quantity or importance in this article, but are used to distinguish one element from another element. The terms "including" and "having" are intended to be inclusive, and represent that in addition to the listed elements, there may be additional elements. When the terms "connected to", "coupled to", etc. are used in this article, an object (e.g., material, element, structure, component, etc.) can be connected to or coupled to another object, regardless of whether the one object is directly connected to or coupled to another object, or whether there are one or more intervening objects between the one object and the other object. In addition, it should be understood that the reference to "one embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the described features.

[0105] In addition to any modifications noted above, those skilled in the art may devise numerous other variations and alternative arrangements without departing from the spirit and scope of the present specification, and the appended claims are intended to cover such modifications and arrangements. Thus, although information enhancement has been described above in detail and in detail in conjunction with what are currently considered to be the most practical and preferred aspects, it is apparent to those of ordinary skill in the art that many modifications, including but not limited to form, function, mode of operation, and use, may be made without departing from the principles and concepts set forth herein. Furthermore, as used herein, these examples and embodiments are intended to be illustrative in all respects only and should not be construed as limiting in any way.

Claims

1. A method for a computing system of a vehicle, the method comprising: In response to an occupant of the vehicle selecting an element of the vehicle's surroundings that is visible to the occupant through a window of the vehicle while the vehicle is moving: capturing, via one or more cameras of the vehicle, a first image of the surrounding environment including the element; identifying the element of the first image; retrieving information about said element from the internet; as well as A second image of the element and the retrieved information about the element are displayed on the window. 2 . The method according to claim 1 , wherein the one or more cameras of the vehicle include at least a front camera, a rear camera, a first side camera installed on a right side of the vehicle, and a second side camera installed on a left side of the vehicle. 3 . The method of claim 1 , wherein the element is selected by the passenger via a touch screen portion of an AR touch screen window.

4. The method of claim 3, wherein identifying the elements of the first image further comprises comparing an estimated line of sight of the passenger from the position of the passenger's eyes through the selected position of the AR touchscreen window to a camera angle of a camera in the one or more cameras of the vehicle. 5 . The method of claim 1 , wherein identifying the element of the first image further comprises determining a location of the element within the surrounding environment using a global positioning system (GPS) of the vehicle.

6. The method of claim 1, wherein identifying the element of the first image further comprises: segmenting the element within the first image; as well as The segmented elements are matched to one or more images of the elements retrieved from one or more local and / or online databases and online reference sites or Internet websites.

7. A method according to claim 6, wherein retrieving information about the element from the Internet also includes retrieving one or more identifiers of the element from the one or more local and / or online databases, reference sites and Internet websites, and performing one or more searches for information about the element in the one or more local and / or online databases, reference sites and Internet websites using the one or more identifiers.

8. The method of claim 7, further comprising performing a first search to retrieve a first set of information; and performing a second search based on elements of the first set of information to retrieve a second set of information. 9 . The method of claim 3 , wherein displaying the second image of the element and the retrieved information about the element on the window further comprises displaying the second image in a portion of the AR touchscreen window.

10. The method of claim 9, further comprising performing an operation on the second image in response to the passenger performing a user interaction with the second image via the AR touchscreen window, the user interaction being one of: Double-click the second image; swiping the second image in a first direction; swiping the second image in a second direction; Touch the second image with two fingers; as well as Slide the passenger's finger across the second image.

11. The method of claim 10, further comprising displaying an internet browser window with information about the element on the AR touch screen window in response to the user interaction, or removing the image in response to the user interaction, or displaying a graphical element in response to the user interaction, thereby allowing the passenger to send the image to one or more other passengers of the vehicle, or displaying one or more graphical elements on the AR touch screen window in response to the user interaction, thereby allowing the passenger to input information and / or images stored on a smart phone and / or stored in the computing system.

12. A computing system for a vehicle, the computing system comprising: In-vehicle AR system; processor; as well as a non-transitory memory storing instructions that, when executed, cause the processor to: In response to a passenger of the vehicle selecting an element of the vehicle's surroundings in an AR touchscreen window of the vehicle coupled to the computing system, the element being visible through the AR touchscreen window while the vehicle is moving: capturing, via one or more cameras of the vehicle, a first image of the surrounding environment including the element; detecting and identifying the elements of the first image; retrieving information about said element from the internet; as well as A second image of the element and the retrieved information about the element are displayed on a portion of the AR touchscreen window.

13. The computing system of claim 12, wherein additional instructions are stored in the non-transitory memory that, when executed, cause the processor to perform an operation on the second image in response to the passenger performing a user interaction with the second image via the AR touchscreen window, the user interaction being one of: Double-click the second image; swiping the second image in a first direction; swiping the second image in a second direction; Touch the second image with two fingers; as well as Slide the passenger's finger across the second image.

14. The computing system of claim 13, wherein the operation is one of: displaying an internet browser window having information about the element on the AR touchscreen window; Displaying a dialog box on the AR touch screen window; Displaying a virtual keyboard on the AR touch screen window; rotating the second image to achieve a 360 degree view of the element; annotating the second image with text, image, audio, video, or different types of content selected by the passenger via one of a microphone, the dialog box, and the virtual keyboard; sharing the second image or the annotated second image with different occupants of the vehicle; storing the second image or the annotated second image in a travel log or a trip planner of the passenger stored in a cloud-based storage device; and The second image or the annotated second image is removed from the AR touchscreen window.

15. The computing system of claim 14, wherein the travel log or trip planner comprises a map-based interface, wherein a link to the second image or the annotated second image is included on the route traveled by the vehicle, and the second image or the annotated second image is displayed when a user selects the link.

Images