Visual representation and control made possible by remote computing and rendering

Abstract

The system comprises: sensors configured to detect gestures relating to a vehicle; a network; an electronic control unit (ECU) connected to the network, a display device, and the sensors, the ECU providing media functions to the vehicle, the ECU being configured to perform the steps of presenting surround view video on the display device, determining gesture states and screen coordinates for the gestures, and transmitting the gesture states and screen coordinates over the network; a camera providing a feed; and an ECU connected to the network and the camera, the ECU providing an advanced driver assistance system (ADAS), the ECU being configured to perform the steps of receiving the gesture states and screen coordinates over the network, generating the surround view video based on the feed from the camera and the gesture states and screen coordinates, and transmitting the surround view video over the network.

Description

Technical Field

[0001] This specification relates to visual representation and control enabled by remote computing and rendering.

Background Art

[0002] In recent years, more vehicles have been equipped with cameras to assist in driving. Initially, a single camera was typically mounted on the rear of the vehicle and directed rearward to assist the driver when reversing. More recent systems use multiple cameras and may be able to generate an artificial view of the vehicle from above. However, such systems may have an inefficient distribution of computational workload and thus have little or no scalability.

Summary of the Invention

[0003] In a first embodiment, the system comprises: a first sensor configured to detect a first gesture relating to a vehicle; a network; a first electronic control unit (ECU) connected to the network, a first display device, and the first sensor, the first ECU providing media functions to the vehicle, and the first ECU being configured to perform the steps of presenting a first surround view video on the first display device, determining a first gesture state and first screen coordinates for the gesture, and transmitting the first gesture state and first screen coordinates over the network; a camera providing a feed; and a second ECU connected to the network and the camera, the second ECU providing an advanced driver assistance system (ADAS), and the second ECU being configured to perform the steps of receiving the first gesture state and first screen coordinates over the network, generating the first surround view video based on the feed from the camera and the first gesture state and first screen coordinates, and transmitting the first surround view video over the network.

[0004] The implementation may include any or all of the following features: The second ECU is configured to perform the steps of: generating a three-dimensional (3D) model using the feed; rendering a two-dimensional (2D) view from the 3D model; and streaming the first surround view video based on the 2D view. The steps of generating the 3D model include: stitching the feed together into a continuous wrap-around 2D view; converting the continuous wrap-around 2D view into a 3D bowl model; and overlaying the 3D model of the vehicle onto the 3D bowl model. The second ECU is configured to encode the 2D view into the first surround view video, and the first ECU is configured to decode the first surround view video before presenting it on the first display device. The first ECU provides a media system for the front occupants of the vehicle, and the vehicle further comprises a third ECU providing a media system for the rear occupants of the vehicle, the third ECU being configured to perform the steps of presenting a second surround view video on a second display device, determining a second gesture state and second screen coordinates for a second gesture detected using a second sensor, and transmitting the second gesture state and second screen coordinates over the network, the second ECU being further configured to receive the second gesture state and second screen coordinates over the network, generating the second surround view video based on the camera feed and the second gesture state and second screen coordinates, and transmitting the second surround view video over the network. The first ECU has a mobile device.The system further includes: a third ECU having a cloud system, wherein the cloud system determines a second gesture state and second screen coordinates for a second gesture detected using a second sensor, and transmits the second gesture state and second screen coordinates over the network, wherein the second ECU is further configured to perform the steps of receiving the second gesture state and second screen coordinates over the network, generating a second surround view video based on the camera feed and the second gesture state and second screen coordinates, and transmitting the second surround view video over the network. The first sensor has touchscreen functionality for the first display device. The first sensor has an infrared camera. The first gesture state and first screen coordinates reflect that the first gesture includes at least one of rotation, translation, or zoom of the first surround view video. The surround view video includes an H.264 stream. The H.264 stream is based on the Realtime Transport Protocol. The H.264 stream is based on the User Datagram Protocol. The system further comprises at least one network switch or gateway that facilitates the transmission of the first gesture state and the first screen coordinates by the first ECU, and the transmission of the first surround view video by the second ECU. The first gesture state and the first screen coordinates are transmitted using a transmission control protocol connection. The first gesture state and the first screen coordinates are transmitted using a real-time streaming protocol connection. The network is an Ethernet® network or an audio-video bridging network.

[0005] In a second embodiment, the method comprises: presenting surround view video on a display device using a first electronic control unit (ECU) connected to a network within a vehicle, the first ECU providing media functions to the vehicle; determining gesture states and screen coordinates for gestures detected using sensors using the first ECU; transmitting the gesture states and screen coordinates via the network using the first ECU; receiving the gesture states and screen coordinates via the network and a second ECU connected to a camera in the vehicle, the second ECU providing an advanced driver assistance system (ADAS); generating the surround view video based on the camera feed and the gesture states and screen coordinates using the second ECU; and transmitting the surround view video via the network using the second ECU.

[0006] The implementation may include any or all of the following features: The first ECU provides a media system for the front occupants of the vehicle. The first ECU has a mobile device. The third ECU comprises a cloud system, and the method further comprises: the cloud system determining a second gesture state and second screen coordinates for a second gesture detected using a second sensor; the cloud system transmitting the second gesture state and second screen coordinates over the network; the second ECU receiving the second gesture state and second screen coordinates over the network; the second ECU generating a second surround view video based on the camera feed and the second gesture state and second screen coordinates; and the second ECU transmitting the second surround view video over the network. The method further comprises recording the surround view video. [Brief explanation of the drawing]

[0007] [Figure 1] This document presents an example of a system that could be implemented for a vehicle to provide visual representation and control made possible by remote computing and rendering.

[0008] [Figure 2] This section outlines examples of visual representations and controls made possible by remote computing and rendering.

[0009] [Figure 3] Figures 3A and 3B show examples of view operations that can be controlled by gestures.

[0010] [Figure 4] An exemplary architecture of a computing device that may be used to implement aspects of this disclosure is illustrated.

[0011] Similar reference numerals in various drawings indicate the same elements. [Modes for carrying out the invention]

[0012] This specification describes examples of systems and techniques that provide a visual representation of a vehicle and its surroundings, and control over that visual representation, made possible by remote computing and rendering. Distributed systems may be used to run and implement surround view monitors with overlaid three-dimensional (3D) renderings provided as a video feed having a 3D rendering application. A surround view monitor system may be provided in the vehicle to present a 360-degree view with a model of the vehicle overlaid on surround video, the video being a stitched-together camera feed view of multiple (e.g., at least front, rear, left, and right outward-facing) cameras of the vehicle. Multiple calibrated video stream synchronization, processes, and 3D computing workloads may be distributed to at least one electronic control unit (ECU), such as an ECU providing an advanced driver assistance system (ADAS) for the vehicle. For example, such an ECU may perform camera feed stitching and 3D model generation and rendering. On the other hand, the gesture processing logic and view presentation workload can be performed by at least a second ECU, such as a media control unit within the vehicle (e.g., an infotainment system), or by a cloud-based application remotely from the vehicle. For example, this could allow the second ECU to instruct the first ECU to display what the user (e.g., a vehicle occupant) requests to see on the surround view monitor. In other words, the computational workload for generating and controlling the surround view representation can be distributed among at least two ECUs, each with distinct roles and functions related to the vehicle. This can improve scalability and facilitate the implementation of multiple different use cases. In some implementations, scalability can allow one or more other applications to be added to ADAS, infotainment systems, and / or cloud systems to enhance adaptability in the surround view monitor, while simultaneously allowing multiple ECUs to use the processed surround view video, while the camera is interfaced on only one ECU.For example, the placement of such additional applications may be favorably guided by locations where the computational workload is expected to be relatively low, without requiring additional wire harnesses or other hardwiring. The distribution of computational workload can be achieved in several ways, including but not limited to using an Ethernet® framework with a transmission control protocol (TCP) connection backbone, which may also be optionally used for other vehicle functions. Other techniques may be used.

[0013] The examples herein refer to vehicles. A vehicle is a machine that transports passengers or cargo, or both. A vehicle may have one or more motors that use at least one type of fuel or other energy source (e.g., electricity). Examples of vehicles include, but are not limited to, cars, trucks, and buses. The number of wheels may differ between types of vehicles, one or more (e.g., all) of the wheels may be used for propulsion of the vehicle, or the vehicle may not have power (e.g., if a trailer is attached to another vehicle). A vehicle may include a passenger compartment that accommodates one or more people. At least one passenger of a vehicle may be considered the driver; in this case, various tools, instruments or other devices may be provided to the driver. In the examples herein, any person transported by a vehicle may be referred to as the “driver” or “passenger” of the vehicle, regardless of whether that person is driving the vehicle, or whether that person has access to the controls for driving the vehicle, or whether that person does not have access to the controls for driving the vehicle. The vehicles in this example are shown for illustrative purposes only, and are depicted as being similar to or identical to one another.

[0014] Examples in this specification refer to ECUs. An ECU is a physical and / or logical unit that performs electronic control of at least one aspect of a vehicle. An ECU may include at least one processor and at least one memory and / or other storage device for instructions of interest (e.g., firmware and / or other software) executed by the processor. An ECU may include at least one internal bus and may support at least one interface for communication by protocol (e.g., automotive bus protocol). For example, an ECU may include at least some of the exemplary components described later with reference to Figure 4. Examples of components or systems that can be controlled by one or more ECUs include, but are not limited to, an engine or other motor (e.g., an inverter for an electric motor); a battery pack or battery module for an electric vehicle; a thermal system; an ADAS or its sensors; or a vehicle infotainment system (e.g., for navigation). For example, an ECU may be designated as a vehicle control unit (e.g., responsible for controlling the powertrain domain). In another example, another ECU may be designated as a vehicle body controller (e.g., responsible for controlling security, access functions, comfort features, and lighting).

[0015] The examples in this specification refer to ADAS. In some implementations, ADAS can perform assisted driving and / or autonomous driving. ADAS can automate one or more dynamic driving tasks at least partially. ADAS may operate in part on the output of one or more sensors, typically located above, below, or inside the vehicle. ADAS may plan one or more trajectories of the vehicle before and / or while controlling the vehicle's movement. Planned trajectories may define the path for the vehicle to travel. Thus, propelling the vehicle according to a planned trajectory may correspond to controlling one or more aspects of the vehicle's operating behavior, including but not limited to the vehicle's steering angle, gear (e.g., forward or reverse), speed, acceleration, and / or braking.

[0016] Autonomous vehicles are an example of ADAS, but not all ADAS are designed to provide fully autonomous vehicles. SAE International defines multiple levels of driver automation, commonly referred to as Level 0, 1, 2, 3, 4, and 5. For example, a Level 0 system or driving mode does not require continuous vehicle control by the system. For example, a Level 1 system or driving mode may include adaptive cruise control, emergency brake assist, automatic emergency brake assist, lane keeping, and / or lane centering. For example, a Level 2 system or driving mode may include highway assist, autonomous obstacle avoidance, and / or autonomous parking. For example, a Level 3 or 4 system or driving mode may include progressively increasing control of the vehicle by the driver assistance system. For example, a Level 5 system or driving mode may not require human intervention in the driver assistance system.

[0017] The examples herein refer to sensors. A sensor is configured to detect one or more aspects of its environment and to output a signal that reflects such detection. The detected aspects may be static or dynamic at the time of detection. For illustrative purposes only, a sensor may indicate one or more of the following: the distance between the sensor and an object, the speed of a vehicle carrying the sensor, the trajectory of the vehicle, or the acceleration of the vehicle. A sensor may generate an output without probing its surroundings using anything (e.g., passive detection, such as an image sensor that captures electromagnetic radiation), or a sensor may probe its surroundings (e.g., active detection by emitting electromagnetic radiation and / or sound waves) and detect a response to the probe. Examples of sensors that may be used with one or more embodiments include, but are not limited to, optical sensors (e.g., cameras); light-based sensing systems (e.g., optical ranging and detection (LiDAR) devices); radio wave-based sensors (e.g., radar); acoustic sensors (e.g., ultrasonic devices and / or microphones); inertial measuring units (e.g., gyroscopes and / or accelerometers); speed sensors (e.g., for vehicles or their components); position sensors (e.g., for vehicles or their components); orientation sensors (e.g., for vehicles or their components); torque sensors; thermal sensors, temperature sensors (e.g., primary or secondary thermometers); pressure sensors (e.g., for the ambient air or components of a vehicle); humidity sensors (e.g., rain detectors); or seating sensors.

[0018] Figure 1 shows an example of a system 100 that may be implemented with respect to a vehicle to provide visual representation and control made possible by remote calculation and rendering. System 100 may be used with one or more other examples described elsewhere in this specification.

[0019] System 100 includes a front occupant media system 102 that provides media functions for the vehicle. The front occupant media system 102 includes at least one ECU. For example, the front occupant media system 102 may provide or be part of the vehicle's infotainment system. The front occupant media system 102 is connected to at least one display device 104 of System 100. The front occupant media system 102 may present content 106 on the display device 104 using connection 108. For example, content 106 may include a surround view video of the vehicle generated according to any example described herein.

[0020] The forward occupant media system 102 may register gestures created to control surround view video. Here, the gesture 110 is schematically represented as the shape of a dashed line positioned relative to content 106. In some implementations, the display device 104 has a touchscreen function (e.g., based on capacitive or resistive detection) to detect gestures performed on the display device 104. The forward occupant media system 102 may then register the gesture 110 via connection 108. In some implementations, the forward occupant media system 102 includes a sensor 112 (e.g., an infrared camera) capable of detecting the gesture 110. The gesture 110 may be performed by a vehicle occupant (e.g., driver or passenger) or another person (as in the following example relating to interface connection with a mobile device or cloud system). The forward occupant media system 102 may determine the gesture state and screen coordinates corresponding to the gesture 110. For example, screen coordinates may include normalized touch coordinates for gesture 110.

[0021] The forward occupant media system 102 may transmit gesture states and screen coordinates, as indicated by stream 114. Stream 114 may enter or be generated within the vehicle's network. In some implementations, stream 114 is transmitted using a TCP connection of system 100. For example, a publisher-subscriber protocol may be used for the TCP connection to add robustness to the communication channel. In some implementations, stream 114 is transmitted using a real-time streaming protocol (RTSP) connection of system 100.

[0022] Transmitting gesture states and screen coordinates by one ECU (here, the forward occupant media system 102) allows another ECU to perform other processes for surround view video. This enables a favorable distribution of the workload for computational tasks within system 100 related to surround view video. Some vehicles may have more of their ECUs in one area than in other areas, and therefore have more processing power. Workload distribution, therefore, can enable more efficient use of computational resources and facilitate better scalability for enhanced functionality. Furthermore, distributing gesture states and screen coordinates over a network avoids the installation of additional communication harnesses, which would significantly increase the material cost and manufacturing effort of the vehicle, as well as its unloaded weight. Thus, this subject can provide adaptability to move task execution and functions freely around the vehicle and avoid the concentration of any function in any individual ECU.

[0023] Stream 114 can be sent to network switch / gateway 116-1. The system may include N network switches / gateways where N = 1, 2, …, and network switches / gateways 116-1 and 116-N are shown. Network switches / gateways 116-1 through 116-N are interconnected by a wired or wireless connection 118 and may operate based on one or more processors and may function as gateways using a combination of software and hardware. For example, one or more of network switches / gateways 116-1 through 116-N can be a vehicle's ECU or vice versa. One or more of network switches / gateways 116-1 through 116-N can be, for example, a switch or a gateway having a switch.

[0024] In some implementations, the vehicle's network is an Ethernet® network and network switches / gateways 116-1 through 116-N are Ethernet® gateways or Ethernet® switches (e.g., arranged in a ring topology). For example, network switches / gateways 116-1 through 116-N can perform message encapsulation and decapsulation, read tags (such as those for non-Ethernet® communications like virtual local area networks) and label messages with them, and route Ethernet® packages. As such, network switches / gateways 116-1 through 116-N can facilitate communication within the vehicle and / or with external systems. Other approaches can be used. For example, system 100 can include an audio video bridging network.

[0025] The system may include an ADAS 120 for a vehicle. The ADAS 120 may be configured to execute any of the ADAS functions 122. The ADAS 120 may use sensor outputs from one or more sensors to perform its tasks including, but not limited to, those from a set of cameras 124 each providing a video feed 126 to the ADAS 120. As illustrated below, the ADAS 120 may use the video feed in the execution of distributed computing tasks for surround view video and may also be used for other purposes. The ADAS 120 may be able to calibrate each of the cameras 124 and / or otherwise control the feed 126.

[0026] The ADAS 120 may be implemented based on a range of physical and / or logical components. Examples include, but are not limited to: a recognition component that receives sensor data and performs segmentation; object detection, classification, and tracking; vehicle localization; sensor fusion; motion planning; motion planning; prediction; trajectory construction; and vehicle actuation (e.g., by steering, gear selection, acceleration, and / or braking). Other functions may be used additionally or alternatively. The surround view video may be provided, to name just one example, for the parking assist feature of the ADAS 120.

[0027] The ADAS 120 may participate in the workload distribution of the system 100. In some implementations, the ADAS 120 includes components 128 for multi-video synchronization, processing, and 3D calculations. The components 128 may facilitate the ADAS 120 receiving a stream 130 of gesture states and screen coordinates based on a stream 114 generated by (e.g., the front passenger media system 102) via a network; generating a surround view video based on the feeds 126 of the cameras 124 and the received gesture states and screen coordinates; and transmitting the surround view video via a network.

[0028] Such network communication with ADAS120 may be performed using a communication component 132 (e.g., an interface to one or more of the network switches / gateways 116-1 to 116-N). Surround view video may be transmitted in stream 134, which may have different characteristics from streams 114 or 130. In some implementations, stream 134 is an H.264 stream. The H.264 stream may be based on the Real-Time Transport Protocol (RTP). For example, this may facilitate error concealment in the event of packet loss. The H.264 stream may be based on the User Datagram Protocol (UDP).

[0029] The forward occupant media system 102 may receive a stream 136 (for example, based on a stream 134 from the ADAS 120) and present surround view video as content 106 on the display device 104. For example, this may include decoding the stream 136 (for example, an H.264 stream).

[0030] The techniques described herein may offer advantageous scalability for surround view video. In some implementations, the vehicle of system 100 may include a rear passenger media system 138. For example, the rear passenger media system 138 may be connected to a display device 140 by a connection 142, and the display device 140 may be designated in the rear row of seats in the vehicle (e.g., the second or third row). System 100 may enable the generation and control of surround view video using the rear passenger media system 138 in a similar manner to that of the front passenger media system 102. The rear passenger media system 138 may register generated gestures 144 for content 146. Gestures 144 may be detected using the touchscreen functionality of the display device 140 and / or sensor 148 (e.g., an infrared camera). The rear passenger media system 138 may determine the gesture state and screen coordinates corresponding to the gesture 144 and transmit them to the network as a stream 150. After processing by ADAS120, the surround view video stream 152 can be received by the rear occupant media system 138. That is, the rear occupant media system 138 illustrates another example of one ECU in system 100 that distributes the computational tasks to another ECU (for example, to ADAS120).

[0031] In some implementations, system 100 may include a mobile device 154. In some implementations, the mobile device 154 may be a smartphone, tablet, smartwatch, or other wearable device. For example, the mobile device 154 is used by the vehicle owner or another person to view and interact with a surround view video of the vehicle. The mobile device 154 has a display device 156 on which a gesture 158 is generated for content 160. The gesture 158 may be detected using the touchscreen functionality of the display device 156 and / or sensors such as an infrared camera. The mobile device 154 may determine the gesture state and screen coordinates corresponding to the gesture 158 and transmit them to a network as a stream 162. For example, the stream 162 is generated using a wireless transmission device. After processing by ADAS 120, the stream 164 of the surround view video (e.g., via a wireless signal) may be received by the mobile device 154. In other words, the mobile device 154 illustrates another example of one ECU in system 100 that distributes computing tasks to another ECU (for example, to ADAS 120).

[0032] In some implementations, system 100 may include a cloud system 166. In some implementations, the cloud system 166 may include one or more processor-based devices or systems that are remote from the vehicle of system 100. The cloud system 166 may control an application that provides surround view video. Thus, the cloud system 166 may determine gesture states and screen coordinates corresponding to gestures and transmit them to the network of system 100 as part of stream 168 (e.g., at least partially wirelessly). After processing by ADAS 120, the surround view video may be received by the cloud system 166 as part of stream 168. This may allow the cloud system 166 to control and / or record the surround view video. That is, the cloud system 166 illustrates another example of one ECU in system 100 distributing computational tasks to another ECU (e.g., ADAS 120).

[0033] Surround view video may be used for one or more purposes. In some implementations, system 100 may facilitate surround view recording. For example, this may allow occupants to capture scenic views while driving or when the vehicle is stopped. Commands to start or stop recording, or other recording control operations, may be generated by gestures along the lines of other examples described herein. Thus, recording may be started by any of the forward occupant media system 102, the rear occupant media system 138, the mobile device 154, and / or the cloud system 166. Recorded media may be stored in at least one component of system 100.

[0034] In some implementations, at least one of the cameras 124 may be located inside the vehicle's cabin. Camera 124 may capture images of one or more occupants. These images may be integrated into a surround-view video, or otherwise combined. For example, this may allow images of occupants to appear together with images of the vehicle's surrounding environment, such as scenic views.

[0035] Figure 2 schematically illustrates an example 200 of visual representation and control made possible by remote computing and rendering. Example 200 can be used in conjunction with one or more other examples described elsewhere in this specification. Example 200 includes a component 128 that performs an action conceptually illustrated on the left side of Example 200, and a media system 202 that performs an action conceptually illustrated on the right side of Example 200. For example, the media system 202 may include the forward occupant media system 102, the rear occupant media system 138, the mobile device 154, and / or the cloud system 166 in Figure 1. Example 200 includes several actions exemplified here as actions 204-226.

[0036] Operations 204-218 may be performed by component 128. In operation 204, component 128 may interface with vehicle camera feeds 126. In some implementations, this includes rear, front, left, and right views. For example, one or more of the feeds 126 may be fisheye camera feeds.

[0037] In operation 206, component 128 may stitch together feeds 126 into a continuous wrap-around two-dimensional (2D) view 228. Generating the continuous wrap-around 2D view 228 involves determining where the contents of the individual feeds 126 meet each other so that continuity is guaranteed. For example, the continuous wrap-around 2D view 228 may contain four 2D views arranged orthogonally in a rectangular shape.

[0038] In operation 208, component 128 may convert a continuous wraparound 2D view 228 into a 3D bowl model 230. For example, this may involve mapping and / or interpolating the contents of the continuous wraparound 2D view 228 into a non-orthogonal shape.

[0039] In operation 210, component 128 may overlay the 3D model 232 of the vehicle onto the 3D bowl model 230. For example, the 3D model 232 may be placed in the center of the 3D bowl model 230.

[0040] In operation 212, component 128 may control the virtual viewpoint 234 of the 3D bowl model 230 and 3D model 232 according to gesture states and screen coordinates received from other ECUs (e.g., from media system 202). That is, the virtual viewpoint 234 determines a specific view to be calculated from the 3D arrangement at any given time.

[0041] In operation 214, component 128 may render a 2D view 236 of the 3D bowl model 230 and the 3D model 232 according to the virtual viewpoint 234.

[0042] In operation 216, component 128 may encode the 2D view 236 according to code 238. In some implementations, the 2D view 236 may be encoded in H.264 format. For example, encoding can reduce the required network bandwidth compared to unencoded transmission.

[0043] In operation 218, component 128 may transmit (optionally encode) the 2D view 236 over the network. For example, the 2D view 236 may be transmitted in H.264 format over Ethernet® based on RTP or UDP.

[0044] Operations 220-226 may be performed by media system 202. In operation 220, media system 202 may decode the 2D view 236 according to code 238. For example, the 2D view 236 may be decoded from H.264 format.

[0045] In operation 222, the media system 202 may present surround view video on the display device 240. The media system 202 may detect gestures performed in relation to the surround view video, as schematically illustrated by arrow 242.

[0046] In operation 224, the media system 202 may execute gesture calculation logic 244 on the detected gesture to determine the gesture state and screen coordinates. For example, the touch event of the gesture may be evaluated.

[0047] In operation 226, the media system 202 may transmit gesture states and screen coordinates over the network. For example, a TCP or RTSP connection may be used. This allows component 128 to base its rendering of the 2D view 236 on the gesture states and screen coordinates in order to generate the requested surround view video.

[0048] Figures 3A and 3B illustrate examples of view operations that can be controlled by gestures. The examples are described with reference to display device 300 and may be used in conjunction with one or more other examples described elsewhere in this specification. For example, display device 300 may be any or all of display device 240 (Figure 2) or display devices 104, 140, or 156 (Figure 1).

[0049] In Figure 3A, view 302 is presented on the display device 300. View 302 shows a 3D model 304 of a vehicle. The 3D model 304 is currently shown in a top view (i.e., from above) as part of a surround view video showing the vehicle and its surroundings. The user may perform one or more gestures to control the surround view video. Gestures may include, but are not limited to, panning, double tapping, pinching, or swiping. Arrows 306A and 306B schematically illustrate that view 302 can be translated by gestures. For example, the user may move the 3D model 304 and its surroundings linearly by performing a first gesture. Arrow 308 schematically illustrates that view 302 can be rotated by gestures. In some implementations, the user may rotate the 3D model 304 and its surroundings by performing a second gesture. In some implementations, the 3D model 304 may remain in a top view after rotation, or the 3D model may rotate away from the top view to a different view (e.g., an oblique view). For example, the rotation may be user-specified or pre-set. Arrow 310 schematically illustrates that the view 302 may be zoomed by a gesture. In some implementations, the user may zoom in on the 3D model 304 and its periphery (e.g., make them larger or smaller) by performing a third gesture.

[0050] In Figure 3B, view 312 is presented on the display device 300. View 312 shows a perspective view of a 3D model 304 of a vehicle as part of a surround view video showing the vehicle and its surroundings. The user can perform one or more gestures to control the surround view video. Arrows 306A and 306B schematically illustrate that view 312 can be translated by a first gesture. Arrow 308 schematically illustrates that view 312 can be rotated by a second gesture. For example, the 3D model 304 and its surroundings may remain in the perspective view after rotation, or may be rotated out of the perspective view to a different view. Arrow 310 schematically illustrates that view 312 can be zoomed by a third gesture.

[0051] Figure 4 shows an exemplary architecture of a computing device 400 that may be used to implement aspects of this disclosure, including any of the systems, apparatus and / or techniques described herein, or any other systems, apparatus and / or techniques that may be used in various possible embodiments.

[0052] The computing device shown in Figure 4 can be used to run the operating systems, application programs, and / or software modules (including software engines) described herein.

[0053] In some embodiments, the computing device 400 includes at least one processing device 402 (e.g., a processor), such as a central processing unit (CPU). Various processing devices are available from various manufacturers, e.g., Intel or Advanced Micro Devices. In this example, the computing device 400 also includes system memory 404 and a system bus 406 that connects various system components, including the system memory 404, to the processing device 402. The system bus 406 is one of any number of bus structures that may be used, including but not limited to a memory bus or memory controller; peripheral bus; and local bus, using any of various bus architectures.

[0054] Examples of computing devices that may be implemented using computing device 400 include desktop computers, laptop computers, tablet computers, mobile computing devices (such as smartphones, touchpad mobile digital devices, or other mobile devices), or other devices configured to process digital instructions.

[0055] The system memory 404 includes a read-only memory 408 and a random-access memory 410. A basic input / output system 412, which includes basic routines that function to transfer information within the computing device 400 during startup, may be stored in the read-only memory 408.

[0056] The computing device 400 also includes, in some embodiments, a storage device 414, such as a hard disk drive, for storing digital data. The storage device 414 is connected to the system bus 406 by a storage interface 416. The storage device 414 and its associated computer-readable medium provide non-volatile and non-transient storage of computer-readable instructions (including application programs and program modules), data structures, and other data for the computing device 400.

[0057] The exemplary environments described herein employ hard disk drives as storage devices, but other types of computer-readable storage media are used in other embodiments. Examples of these other types of computer-readable storage media include magnetic cassettes, flash memory cards, solid-state drives (SSDs), digital video discs, Bernoulli cartridges, compact disk read-only memory, digital versatile disk read-only memory, random access memory, or read-only memory. Some embodiments include non-temporary media. For example, computer program products may be tangibly embodied within non-temporary storage media. Additionally, such computer-readable storage media may include local storage or cloud-based storage.

[0058] Numerous program modules may be stored in the storage device 414 and / or system memory 404, including an operating system 418, one or more application programs 420, other program modules 422 (such as the software engine described herein), and program data 424. The computing device 400 may utilize any suitable operating system.

[0059] In some embodiments, the user provides input to the computing device 400 through one or more input devices 426. Examples of input devices 426 include a keyboard 428, a mouse 430, a microphone 432 (e.g., for voice and / or other audio input), a touch sensor 434 (such as a touchpad or touch-sensitive display), and a gesture sensor 435 (e.g., for gesture input). In some implementations, the input devices 426 provide detection based on presence, proximity, and / or motion. Other embodiments include other input devices 426. The input devices may be connected to the processing device 402 through input / output interfaces 436 coupled to the system bus 406. These input devices 426 may be connected by any number of input / output interfaces, such as parallel ports, serial ports, game ports, or a universal serial bus. In some possible embodiments, wireless communication between the input device 426 and the input / output interface 436 is also possible, and to name a few examples, these include infrared, Bluetooth® wireless technology, 802.11a / b / g / n, cellular, ultra-wideband (UWB), ZigBee®, or other radio frequency communication systems.

[0060] In this exemplary embodiment, a display device 438, such as a monitor, liquid crystal display device, light-emitting diode display device, projector, or touch-sensitive display device, is also connected to the system bus 406 via an interface such as a video adapter 440. In addition to the display device 438, the computing device 400 may include various other peripheral devices (not shown), such as speakers or printers.

[0061] The computing device 400 may be connected to one or more networks through a network interface 442. The network interface 442 may provide wired and / or wireless communication. In some implementations, the network interface 442 may include one or more antennas for transmitting and / or receiving wireless signals. When used in a local area networking environment or a wide area networking environment (such as the Internet), the network interface 442 may include an Ethernet® interface. Other possible embodiments use other communication devices. For example, some embodiments of the computing device 400 include a modem for communication across the network.

[0062] The computing device 400 may include at least some form of computer-readable medium. The computer-readable medium includes any available medium that can be accessed by the computing device 400. For example, the computer-readable medium includes computer-readable storage medium and computer-readable communication medium.

[0063] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented within any device configured to store information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random-access memory, read-only memory, electrically erasable programmable read-only memory, flash memory, or other memory technologies, compact disk read-only memory, digital versatile disk or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media that can be accessed by computing device 400 and used to store desired information.

[0064] Computer-readable communication media typically include any information transmission medium that embodies computer-readable instructions, data structures, program modules, or other data within modulated data signals, such as carrier waves or other transmission mechanisms. The term “modulated data signal” refers to a signal in which one or more of its characteristics are set or modified in a manner that encodes information within the signal. Examples of computer-readable communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. Any combination of the above also falls within the scope of computer-readable media.

[0065] The computing device shown in Figure 4 is also an example of a programmable electronic device that may include one or more such computing devices, and when multiple computing devices are included, such computing devices can be coupled together to a suitable data communication network to collectively perform various functions, methods or operations disclosed herein.

[0066] In some implementations, the computing device 400 may be characterized as an ADAS computer. For example, the computing device 400 may include one or more components that may be used to handle tasks arising in the field of artificial intelligence (AI). The computing device 400 then includes sufficient power and necessary support architecture for the demands of ADAS or AI in general. For example, the processing device 402 may include a multi-core architecture. As another example, the computing device 400 may include one or more coprocessors in addition to, or as part of, the processing device 402. In some implementations, at least one hardware accelerator may be coupled to the system bus 406. For example, a graphics processing unit may be used. In some implementations, the computing device 400 may implement neural network-specific hardware to handle one or more ADAS tasks.

[0067] The terms “substantially” and “about” as used throughout this specification are used to describe and account for small variations, such as those resulting from processing variability. For example, they may mean less than or equal to ±5%, for example less than or equal to ±2%, for example less than or equal to ±1%, for example less than or equal to ±0.5%, for example less than or equal to ±0.2%, for example less than or equal to ±0.1%, for example less than or equal to ±0.05%. Also, as used herein, indefinite articles such as “a” or “an” mean “at least one.”

[0068] It should be understood that all combinations of the concepts described above and any additional concepts discussed in more detail below (provided that such concepts are not mutually contradictory) are intended to be part of the subject matter of the invention disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are intended to be part of the subject matter of the invention disclosed herein.

[0069] We have described several implementations. Nevertheless, it should be understood that various modifications may be made without deviating from the intent and scope of this specification.

[0070] Furthermore, the logical flow shown in the diagram does not require a specific or sequential order to achieve the desired result. In addition, other processes may be provided, or processes may be excluded from the described flow; other components may be added to the described system, or other components may be removed from the described system. Therefore, other implementations fall within the scope of the following claims.

[0071] While specific features of the described implementations have been shown as described herein, many modifications, substitutions, alterations, and equivalents will now come to mind for those skilled in the art. It should be understood that the appended claims are intended to encompass all such modifications and alterations that fall within the scope of these implementations. They are presented merely as examples and not as limitations, and it should be understood that various modifications may be made in form and detail. Any parts of the apparatus and / or methods described herein may be combined in any combination, except for mutually exclusive combinations. The implementations described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different implementations described.

Claims

1. A first sensor configured to detect a first gesture related to a vehicle; network; The network, the first display device, and the first sensor are connected to a first electronic control unit (ECU), the first ECU providing media functions to the vehicle, and the first ECU is configured to perform the steps of presenting a first surround view video on the first display device, determining a first gesture state and first screen coordinates for the first gesture, and transmitting the first gesture state and first screen coordinates via the network; A camera that provides a feed; and A second ECU is connected to the network and the camera. The second ECU provides the vehicle with an advanced driver assistance system (ADAS). The second ECU is configured to perform the following steps: receive the first gesture state and the first screen coordinates via the network; generate the first surround view video based on the camera feed, the first gesture state, and the first screen coordinates; and transmit the first surround view video via the network. A system that includes these features.

2. The aforementioned second ECU is: A procedure for generating a three-dimensional (3D) model using the aforementioned feed; A procedure for rendering a two-dimensional (2D) view from the aforementioned 3D model; and A procedure for streaming the first surround view video based on the 2D view. The system according to claim 1, configured to perform the following:

3. The procedure for generating the aforementioned 3D model is as follows: Procedure for connecting the aforementioned feeds into a continuous wrap-around 2D view; A procedure for converting the aforementioned continuous wraparound 2D view into a 3D bowl model; and Procedure for overlaying the 3D model of the vehicle onto the 3D bowl model. The system according to claim 2, having the following features.

4. The system according to claim 2, wherein the second ECU is configured to encode the 2D view into the first surround view video, and the first ECU is configured to decode the first surround view video before presenting the first surround view video on the first display device.

5. The system according to claim 1, wherein the first ECU provides a media system for the front occupants of the vehicle, and the vehicle further comprises a third ECU providing a media system for the rear occupants of the vehicle, the third ECU being configured to perform the steps of presenting a second surround view video on a second display device, determining a second gesture state and second screen coordinates for a second gesture detected using a second sensor, and transmitting the second gesture state and second screen coordinates over the network, the second ECU being further configured to perform the steps of receiving the second gesture state and second screen coordinates over the network, generating the second surround view video based on the feed from the camera and the second gesture state and second screen coordinates, and transmitting the second surround view video over the network.

6. The system according to claim 1, wherein the first ECU has a mobile device.

7. The system according to claim 1, further comprising: a third ECU having a cloud system, wherein the cloud system determines a second gesture state and a second screen coordinate for a second gesture detected using a second sensor, and transmits the second gesture state and the second screen coordinate via the network, wherein the second ECU is further configured to perform the steps of receiving the second gesture state and the second screen coordinate via the network, generating a second surround view video based on the feed from the camera and the second gesture state and the second screen coordinate, and transmitting the second surround view video via the network.

8. The system according to claim 1, wherein the first sensor has a touchscreen function for the first display device.

9. The system according to claim 1, wherein the first sensor has an infrared camera.

10. The system according to claim 1, wherein the first gesture state and the first screen coordinates reflect that the first gesture includes at least one of rotation, translation, or zoom of the first surround view video.

11. The system according to claim 1, wherein the first surround view video includes an H.264 stream.

12. The H.264 stream is based on a real-time transport protocol, as described in claim 11.

13. The H.264 stream is based on the user datagram protocol, as per the system of claim 11.

14. The system according to any one of claims 1 to 13, further comprising at least one network switch or gateway that facilitates the transmission of the first gesture state and the first screen coordinates by the first ECU and the transmission of the first surround view video by the second ECU.

15. The system according to any one of claims 1 to 13, wherein the first gesture state and the first screen coordinates are transmitted using a transmission control protocol connection.

16. The system according to any one of claims 1 to 13, wherein the first gesture state and the first screen coordinates are transmitted using a real-time streaming protocol connection.

17. The system according to any one of claims 1 to 13, wherein the network is an Ethernet® network or an audio-video bridging network.

18. In the step of presenting surround view video on a display device using a first electronic control unit (ECU) connected to the vehicle's network, the first ECU provides media functions to the vehicle; The first step is to determine the gesture state and screen coordinates for the gesture detected using the sensor, using the first ECU; The first ECU transmits the gesture state and the screen coordinates via the network; In the step of receiving the gesture state and the screen coordinates by a second ECU connected to the network and the vehicle's camera, the second ECU provides the vehicle with an advanced driver assistance system (ADAS); The second ECU generates the surround view video based on the camera feed, the gesture state, and the screen coordinates; and The second ECU transmits the surround view video via the network. A method that includes [a certain feature].

19. The method according to claim 18, wherein the first ECU provides a media system for the front occupants of the vehicle.

20. The method according to claim 18, wherein the first ECU has a mobile device.

21. The third ECU is equipped with a cloud system, and the method further: The cloud system determines a second gesture state and second screen coordinates for a second gesture detected using a second sensor; The cloud system transmits the second gesture state and the second screen coordinates via the network; The second ECU receives the second gesture state and the second screen coordinates via the network; The second ECU generates a second surround view video based on the feed of the camera, the second gesture state, and the second screen coordinates; and The second ECU transmits the second surround view video via the network. The method according to claim 18, comprising:

22. The method according to any one of claims 18 to 21, further comprising the step of recording the surround view video.

Images