Methods and Systems for Augmented Reality in Automotive Applications

Abstract

A method for augmented reality in an automotive application includes acquiring an image frame from a camera in the vehicle, identifying at least one environmental object in the image frame, determining augmented content based on the at least one environmental object, rendering an augmented image frame based on the image frame and the augmented content, and displaying the augmented image frame to a user.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to methods and systems for augmented reality in automotive applications.

Background Art

[0002] Automobiles are increasingly equipped with a large number of sensors. Such sensors include, for example, cameras, LiDAR sensors, ultrasonic sensors, and the like. These sensors can support the implementation of driving assistance systems and / or autonomous driving. Other applications may also benefit from the availability of signals obtained from these sensors. The availability of signals provided by these sensors may enable the realization of other applications, for example, applications that have not directly received such signals conventionally.

Summary of the Invention

Problems to be Solved by the Invention

[0003] This summary is provided to introduce a selection of concepts that are further described in the following forms for implementing the invention. This summary is not intended to identify the main features or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter.

Means for Solving the Problems

[0004] Generally, in one aspect, an embodiment relates to a method for augmented reality in automotive applications, the method including obtaining an image frame from a camera of an automobile, identifying at least one environmental object in the image frame, determining augmented content based on the at least one environmental object, rendering an augmented image frame based on the image frame and the augmented content, and displaying the augmented image frame to a user.

[0005] Generally, in one embodiment, the embodiment relates to a system for augmented reality in an automotive application, the system comprising: an environment interpretation engine that identifies at least one environment object in an image frame acquired from a camera of the vehicle; a content augmentation engine that determines augmented content based on the at least one environment object; and a rendering engine that renders an augmented image frame based on the image frame and the augmented content in order to display the augmented image to a user.

[0006] Generally, in one embodiment, the embodiment relates to a non-temporary computer-readable medium (CRM) for storing computer-readable program code for augmented reality in an automotive application, wherein the computer-readable program code causes a computer system to acquire an image frame from a camera in the vehicle, identify at least one environmental object in the image frame, determine augmented content based on the at least one environmental object, render an augmented image frame based on the image frame and the augmented content, and display the augmented image frame to a user.

[0007] Other aspects and advantages of the claimed subject matter will become apparent from the following description and the attached claims. [Brief explanation of the drawing]

[0008] Specific embodiments of the disclosed technology will be described in detail with reference to the accompanying drawings. Similar elements in each drawing will be given the same reference numerals for consistency.

[0009] [Figure 1A] This document presents one or more augmented reality (AR) / virtual reality (VR) scenarios for automobiles according to one or more embodiments. [Figure 1B] This document presents one or more augmented reality (AR) / virtual reality (VR) scenarios for automobiles according to one or more embodiments. [Figure 1C]This document describes a system for augmented reality in automotive applications according to one or more embodiments. [Figure 2] This document describes a method for augmented reality in automotive applications according to one or more embodiments. [Figure 3] One or more embodiments of a method (300) for post-processing identified environmental objects are shown. [Figure 4] This example shows how to consolidate multiple blobs associated with a single environment object into a single blob. [Figure 5] Examples of one or more embodiments are shown. [Figure 6] A computer system according to an embodiment of this disclosure is shown. [Modes for carrying out the invention]

[0010] In the detailed description of embodiments of the present disclosure below, numerous specific details are provided to give a more complete understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be implemented without these specific details. In other examples, well-known features are not described in detail to avoid unnecessarily complicating the description.

[0011] Throughout this application, ordinal numbers (e.g., 1st, 2nd, 3rd, etc.) may be used as adjectives for elements (i.e., any noun in this application). The use of ordinal numbers is not intended to imply or create a particular order of elements, nor is it intended to limit any element to only a single element unless explicitly disclosed, such as by using terms like "before," "after," or "single." Rather, the use of ordinal numbers is for distinguishing between elements. For example, the first element is different from the second element, which may encompass multiple elements and may come after (or before) the second element in the order of elements.

[0012] Generally, embodiments of this disclosure include systems and methods for augmented reality in automotive applications. Virtual reality (VR) and augmented reality (AR) are increasingly used in entertainment, education, surveillance and supervision, control, and other applications. VR and AR applications frequently require input from sensors, such as cameras. In one or more embodiments of this disclosure, a platform is provided that leverages automotive sensors to deliver VR and AR applications to automotive users. For passengers, these applications may provide entertainment and / or informational content. For drivers, these applications may provide informational and / or assistive content. An example of an AR application according to an embodiment of this disclosure is a video game that utilizes image frames acquired using cameras associated with the automotive to provide real-time content of the real world in the video game. These and other applications, as well as the underlying platform according to embodiments of this disclosure, will be described with reference to the figures below.

[0013] Figures 1A and 1B schematically illustrate automotive AR / VR scenarios (100) according to one or more embodiments. Figure 1A shows a scenario that includes cloud-based components, while Figure 1B shows a scenario that does not necessarily require cloud-based components.

[0014] Referring to Figure 1A, in the shown scenario (100), the automobile (196) is in the environment (198). The automobile may be, for example, a passenger car, a bus, or any other type of vehicle. A driver and / or passengers may be present inside the automobile. The environment (198) is the environment surrounding the automobile. In a typical scenario, the environment (198) may include elements such as roads, other traffic including other vehicles, pedestrians, and buildings. The automobile (196) may move within the environment (198). Therefore, the environment (198) may be transient with respect to the automobile (196).

[0015] In one or more embodiments, the automobile is equipped with one or more cameras (110). One or more cameras (110) capture image frames of the environment (198). These image frames may be processed and displayed on a display (160). The display (160) may be the display of a portable user device such as a smartphone or tablet. In one or more embodiments, the image frames are expanded before being displayed on the display (160). The type of expansion may be application-dependent. For example, a video game may include elements different from those of an educational application. A detailed description is provided below with reference to Figure 1C illustrating the system (190) and Figures 2 and 3 illustrating the methods (200, 300).

[0016] In scenario (100), the exchange of image frames (e.g., from camera (110) to display (160)) may be performed via the cloud environment (170). Extensions and / or other operations may be performed in the cloud environment (170). Additional details are provided below with reference to the flowcharts in Figure 1C and Figures 2 and 3.

[0017] In one or more embodiments, a developer kit is provided that includes an in-cloud interface (172) and an in-vehicle instance (162) to facilitate the development of possible AR / VR applications. The developer kit may include tools for processing image frames and / or other sensor data acquired in either raw or pre-processed form. The developer kit may further provide a developer interface for developers and / or users to access and further use (e.g., by a game engine) the image frames and / or other sensor data. The developer interface may be provided in the form of one or more application programming interfaces (APIs). The APIs may exist to provide access to sensor data such as, for example, image frames acquired from a camera, LiDAR data, vehicle speed, steering angle, Global Positioning System (GPS) data, and Inertial Measurement Unit (IMU) data. The APIs may further exist for a 3D development space and rendering of 2.5D output. The APIs may provide access to any of the functions described below.

[0018] The developer interface may be standardized for easy access to this data for a wide variety of applications that use this data. The developer interface may also provide access to data that is the result of machine learning-based processing of this data, as well as data acquired from the camera (110) and / or other sensors, for example, as illustrated with reference to the flowcharts in Figures 2 and 3. The developer kit may enable third parties, such as application developers, to provide content to run on the system, as further illustrated with reference to Figure 1. The availability of data in a standardized format through the developer kit can simplify the task of application development and enable non-experts to become VR / AR content creators.

[0019] In the embodiment shown in FIG. 1A, the in-cloud interface (172) is a component of the developer kit and may be accessible by an application developer, for example, for application testing, deployment, etc. The in-vehicle instance (162) is a component of the developer kit and is locally executed, for example, on a user input device or a display device. The in-vehicle instance (162) may collect user input commands, perform operations related to the visualization of content displayed to the user, and / or execute an application.

[0020] Moving on to FIG. 1B, in the scenario (102) shown, the vehicle (196) is within an environment (198). The scenario (102) is similar to the scenario (100) but does not depend on a cloud environment. The developer kit in the scenario (102) includes in-vehicle computing (174) and an end-user interface (164). The in-vehicle computing (174) and the end-user interface (164) are local implementations without cloud processing but may be functionally equivalent to the in-cloud interface (172) and the in-vehicle instance (162), respectively. Periodic communication with other remote (e.g., cloud-based components) may be limited to, for example, application downloads, application service provision, etc.

[0021] FIG. 1C is a block diagram of a system (190) for augmented reality in a vehicle application according to one or more embodiments. In one or more embodiments, the system (190) is associated with a vehicle (196). For example, one or more elements of the system (190) may be components of the vehicle, and the system (190) may be used by a user of the vehicle (e.g., a passenger and / or a driver). Specific scenarios are described below.

[0022] The system includes one or more cameras and / or other sensors (110), an environmental interpretation engine (120), a content extension engine (130), a rendering engine (150), and a display (160). The system may further include one or more user input devices (140). Each of these components will be described later.

[0023] One or more cameras and / or other sensors (110) capture data from the environment (198). The camera (110) may capture an image frame of the environment (198). The image frames may be repeatedly captured at a fixed or adjustable frame rate. The camera (110) may be of any type and may have any field of view, resolution, orientation, etc. In one embodiment, a fisheye camera is used. Data from the camera, such as an image frame, may be provided in any format (e.g., representing RGB pixel values) and may be received by an automotive electronic control unit (ECU). The ECU may include components for video processing by hardware acceleration, including machine learning-based video processing. The ECU may further execute various components of the system (100), such as the environmental interpretation engine (120), the content extension engine (130), and / or the rendering engine (150).

[0024] Other sensors of the vehicle may include, for example, GPS and / or IMU sensors for position and / or orientation tracking, including determination of mapping and localization, vehicle speed, acceleration, etc. Other sensors may further include, for example, LiDAR, radar, and / or ultrasonic sensors, etc., as used for driver assistance functions.

[0025] One or more cameras and / or other sensors (110) may provide image frames, or more generally, sensor data (112). The sensor data (112) may be provided in any form and may depend on the type of sensor. For example, a camera may provide image frames, while a steering angle sensor may provide a value that reflects a measurement of the steering angle.

[0026] In one or more embodiments, sensor data or image frames (112) may be preprocessed as described further below with reference to a flowchart.

[0027] In one or more embodiments, the environment interpretation engine (120) receives sensor data (112), for example in the form of an image frame, and identifies environmental objects (122) within the image frame. For example, roads, pedestrians, other vehicles, buildings, etc., may be identified within the image frame. The environment interpretation engine (120) may perform image processing methods to identify the environmental objects (122). Any type of image processing may be used. For example, the image processing may use a machine learning-based algorithm.

[0028] Machine learning (ML), in a broad sense, is the extraction of patterns and insights from data. The terms “artificial intelligence,” “machine learning,” “deep learning,” and “pattern recognition” are often intricately intertwined, interchangeable, and synonymous throughout the literature. This ambiguity stems from the fact that the field of “extracting patterns and insights from data” has developed simultaneously and independently among numerous classical techniques such as mathematics, statistics, and computer science. For consistency, the terms machine learning, or machine-learned, are used herein. However, those skilled in the art will recognize that the concepts and methods detailed below are not limited by this choice of nomenclature.

[0029] In this specification, the types of machine learning models used for video processing and image frame processing may include, but are not limited to, generalized linear models, Bayesian regression, random forests, and deep models such as neural networks, convolutional neural networks, and recurrent neural networks. The type of machine learning model, whether considered deep or not, is typically associated with additional “hyperparameters” that further describe the model. For example, hyperparameters providing further details about a neural network may include, but are not limited to, the number of layers in the neural network, the choice of activation function, the inclusion of batch normalization layers, and the regularization strength. It should be noted that in the context of machine learning (ML), regularization of a machine learning model refers to the penalty applied to the loss function of the machine learning model and should not be confused with regularization of the earthquake dataset (possible preprocessing step). Generally, in the literature, the selection of hyperparameters surrounding a machine learning model is referred to as the selection of the model “architecture.” Once the type of machine learning model and hyperparameters are selected, the machine learning model is trained to perform the task. According to one or more embodiments, a type of machine learning model and associated architecture are selected, the machine learning model is trained to perform video processing to determine traffic environment image data, the performance of the machine learning model is evaluated, and the machine learning model is used in a production setting (also known as deploying the machine learning model).

[0030] The operations performed by the environmental interpretation engine (120) are described below with reference to the flowchart. The environmental interpretation engine (120) may run on a computer system, for example, as described with reference to Figure 6. In one embodiment, the computer system is an ECU of an automobile.

[0031] In one or more embodiments, the content augmentation engine (130) receives identified environment objects (122) and determines augmented content (132) based on the environment objects (122).

[0032] In one or more embodiments, the enhanced content (132) includes any modifications to the image frame for subsequent display to the user. For example, the enhanced content (132) may include blurring, marking, highlighting, distorting, removing, moving, recoloring, animating, or any other modifications to one or more of the identified environment objects (122). The enhanced content (132) may further include the addition of any number of objects. The added objects may be static or dynamic objects. Static objects may be indicators (e.g., arrows pointing to another object), while dynamic objects may be objects whose properties, such as shape, size, color, position, etc., change over time, for example, from frame to frame. Such dynamic objects may include, for example, animated characters in a video game application, dynamically changing information content, etc.

[0033] In one or more embodiments, the augmented content (132) may be user-controllable, either directly or indirectly. A user input device (140) may enable the user to provide input device commands (142). For example, the user may provide steering commands to control the position, orientation, movement, etc., of an object that is an element of the augmented content (132). Control of the augmented content may be context-specific, and may enable interaction within the augmented content, etc. The user input device (140) enabling control of the augmented content (132) may be a smartphone, game console, tablet, or similar device that enables the user to provide input. The user input device may interface to communicate with the vehicle's computing device (e.g., ECU) using a wired or wireless interface such as WIFI®, USB, etc. In the discussion of possible examples of various applications, a more detailed discussion of different types of augmented content and possible interactions with the augmented content by one or more users is provided below.

[0034] In one or more embodiments, the rendering engine (150) receives sensor data or an image frame (112) and augmented content (132) and generates an augmented image frame (152). In preparing the augmented image frame (152) for display, the rendering engine may perform occlusion detection to render only the pixels that should be visible. For example, if augmented content (132) is present, the rendering engine (150) may generate an object mask, which is then used to identify occlusion and render only the desired pixels of the sensor data or image frame (112). In one or more embodiments, machine learning-generated data is used to generate an object mask, which is then used to identify occlusion and render only the desired pixels. A rendering engine (150) such as Unity or Unreal may be used.

[0035] In one or more embodiments, the display (160) is used to display an extended image frame (152) to the user. The display (160) may be any type of display, such as an in-car screen, a smartphone, a tablet, or a game device display. In one embodiment, the input device (140) and the display (160) are combined into a single device such as a smartphone, a tablet, or a game device.

[0036] System (190) enables various implementations of VR and AR experiences. Several examples are provided below.

[0037] Game Application: The 3D game world may be constructed based on image frames captured by one or more cameras associated with the vehicle. In this case, the environment of the 3D game world is generated in real time based on the actual environment surrounding the vehicle. In one specific example, the vehicle's forward-facing camera may be used to capture video (a series of image frames). The video may be displayed to the user along with various enhancements. The user may steer the cart within this environment using a user input device. The 3D game world may include obstacles to be avoided (e.g., specific environmental objects or virtual objects added as enhancement content). The 3D game world may also include targets (e.g., virtual objects added as enhancement content).

[0038] The 3D game world may support a single user or multiple users to provide a multiplayer experience. While the 3D game world is an AR application, a corresponding VR implementation may be provided by completely replacing all identified environmental objects with augmented content. For example, identified roads may be rendered as artificial roads, and other identified vehicles may be rendered as different forms of vehicles. In this configuration, the video itself may be excluded from the augmented image frames provided as output to the user. The described 3D game world may operate in real time (based on image frames processed in real time) or non-real time (depending on previously recorded image frames).

[0039] Educational Application: The educational environment may be built on image frames captured by one or more cameras associated with the vehicle. The educational environment may be real-time or non-real-time. In one example, the educational application is used to teach traffic rules. For example, the educational application may teach the user the meaning of lane markings and traffic signs, and how to perform certain driving operations such as lane changes.

[0040] Entertainment: The entertainment environment may be built on image frames captured by one or more cameras associated with the vehicle. The entertainment environment may be real-time or non-real-time. In one example, the entertainment application is for a child. Consider the daily drive to and from school. This drive is repetitive. In this example, the user (child) may virtually choose a place (along the road) to grow a virtual plant. The augmented reality experience provided to the user may include all aspects of the plant's life cycle, from sowing seeds to observing growth and harvesting. The plant may also respond to care such as watering and fertilizing. In another example, the environment may be transformed into a Tetris or puzzle game environment based on a real environment.

[0041] A system according to the embodiments of this disclosure may support any of these applications. A user may select a desired application from a set of available applications, or the application may be specified by a third party. For example, a parent may select a specific application for their child.

[0042] Figures 1A, 1B, and 1C show component configurations, but other configurations may be used without departing from the scope of this disclosure. For example, various components may be combined to create a single component. As another example, a function performed by a single component may be performed by two or more components. Various operations performed by the system may be performed on a computing system such as an automotive ECU. At least some of these operations, for example, environment interpretation and rendering, may benefit from the availability of a graphics processing unit (GPU). Therefore, the ECU or other computing system may be equipped with a GPU. Furthermore, although not expressly shown, operations performed by the various components discussed with reference to Figures 1A and 1B may be performed locally or remotely, for example, in a cloud environment (170).

[0043] Figure 2 shows a flowchart of a method for augmented reality in an automotive application according to one or more embodiments.

[0044] The execution of one or more steps in Figure 2 may involve one or more components of the system described in Figure 1. Although the various steps in Figure 2 are presented and described sequentially, those skilled in the art will understand that some or all of the steps may be performed in a different order, combined, or omitted, and that some or all of the steps may be performed in parallel. Furthermore, the steps may be performed actively or passively. The actions that implement the described steps may be represented by electronically readable instructions, such as software code or firmware code, which may be stored in one or more non-temporary media.

[0045] In step 202, sensor data is acquired. In one or more embodiments, the sensor data includes image frames acquired by the camera as described above. In one or more embodiments, the camera is a fisheye camera. Step 202 may include camera-specific preprocessing. For example, the image frames acquired from the camera may be preprocessed to remove distortions such as distortion typical of fisheye cameras. For example, image cropping may be performed to acquire rectangular image frames of content of interest. Any other preprocessing may be performed, such as brightness and / or contrast correction, compression, and resizing. Additional sensor data from other sensors may also be acquired as described above.

[0046] In step 204, environmental objects are identified within image frames. For example, a series of image frames may be analyzed to detect motion. In one or more embodiments, machine learning-based image processing is used for identifying environmental objects. For example, deep learning-based image processing such as OmniDet or Single-Shot Deep MANTA may be used. Training may be performed in advance, for example, based on surround-view fisheye data. The dataset used for training may be environment-specific and may include elements typically encountered when driving a car (e.g., buildings, roads, vehicles, bicycles, animals, pedestrians, etc.). The output obtained as a result of machine learning-based image processing may include depth estimation of environmental objects, semantic segmentation, visual odometry, motion segmentation, and / or object detection. For example, machine learning-generated data from image processing may be used to mark different segments within an image. This information may then be used to generate a navigation path for a character in the scene. The output may be stored and / or transferred as a description of the identified environmental objects. The descriptions of identified environmental objects may be linked to the frame number of the corresponding image frame for synchronization purposes. Additional detections, such as lens dirt detection, may be performed. Examples of environmental objects that can be identified include, but are not limited to, roads, lane markings, curbs, people, riders, vehicles, bicycles, motorcycles, and traffic signs.

[0047] In one or more embodiments, sensor data from other sensors, such as radar, LiDAR, or ultrasonic sensors, may be used to improve the accuracy of detection, depth estimation, and / or semantic segmentation, while GPS and IMU data may be used to determine an accurate estimate of the current vehicle position and / or orientation.

[0048] In step 206, post-processing is performed on the identified environment objects. Post-processing may be performed to obtain data related to the environment objects in a format suitable for input to a rendering engine (e.g., as described in step 210). The rendering engine may expect a description of the environment objects as boundaries for these environment objects. In contrast, the output of the operation in step 204 may be pixel data. In step 206, the data manipulation necessary to obtain the boundaries is performed. Examples of other problems addressed by post-processing include, but are not limited to, false pedestrian clusters, severed pedestrian heads, pedestrians with extraneous background, and undetected road surfaces. Post-processing may be performed on any of the identified environment objects. A description of the post-processing in step 206 is provided below with reference to Figure 3.

[0049] Upon completion of steps 204 and 206, a scene understanding is obtained in which environmental objects are identified based on the performed object detection and classification, and a depth map of the environmental objects becomes available, ensuring that their position, orientation, movement, etc., are known. This includes plane identification, such as horizontal planes established based on road surfaces, vertical planes established based on detected walls or other structures, etc. The obtained scene understanding may govern possible movement in the rendered VR / AR environment as described below.

[0050] In step 208, the augmented content is determined. The augmented content added to the rendered VR / AR environment depends on the intended application of the VR / AR environment.

[0051] Extended content includes modifications to the content within an image frame, such as adding objects and masking.

[0052] The objects that can be added include, for example, one or more avatars, objects, symbols, labels, features, animated characters, text, etc. Objects may be static or dynamic. Dynamic objects may be controlled by the user or by the content extension engine (for example, computer-controlled characters (such as adversaries) moving around in the AR / VR environment). Furthermore, objects may change their behavior based on, for example, context, contact, etc.

[0053] A mask used to modify content within an image frame may include an overlay (e.g., a color filter, a blur filter, etc.), be opaque or partially transparent, and be static or dynamic (e.g., changing properties such as color, contrast, or blinking on / off).

[0054] Objects may be positioned considering the horizontal and vertical planes identified above. For example, the horizontal plane may function as the character's walkable area, while the vertical plane may function as a boundary. Character scaling and path planning may be further performed to generate the character's navigation path. For example, A* or any other path planning algorithm may be used.

[0055] In one or more embodiments, the extended content is linked to the frame number of the corresponding image frame for synchronization purposes.

[0056] In step 210, the augmented image frame is rendered based on the image frame and the augmented content. In an AR environment, rendering may be performed based on inputs including the image frame itself, identified environment objects, and augmented content. In a VR environment, rendering may be performed based on inputs including identified environment objects and augmented content. The description of identified environment objects may include geometry, for example, geometry in the form of boundary pixels for each previously segmented object (e.g., ground, person, vehicle, etc.). Depth information may be included in the description. Similarly, the description of augmented content includes the geometry of the augmented content. Further inputs that may affect rendering may be received from the user. For example, the user may change the rendered view by control commands, steering control actions, etc.

[0057] Rendering takes these inputs and generates pixel position points that can be used to map them to a subsequently displayed 2D scene. In one or more embodiments, rendering considers the geometry of objects (environment objects, extensions) to determine occlusion. Based on the detected occlusion, an invisibility mask is determined based on the contours of the occluding geometry, and rendering is then performed only on the unoccluded portions based on the mask.

[0058] In one or more embodiments, rendering ensures synchronization between image frames, descriptions of identified environment objects, and extended content. Synchronization may be performed based on frame number.

[0059] In step 212, the extended image frame is displayed on the display of a user device, such as a smartphone or tablet display, or on a car display. If rendering is performed for multiple users, for example, who have different views, the extended image frame may be displayed on multiple displays.

[0060] Steps 202-212 may be performed in a loop, for example, based on a fixed frame rate. This frame rate may correspond to or differ from the camera's frame rate.

[0061] Figure 3 shows a method (300) for post-processing identified environmental objects according to one or more embodiments. For the execution of method (300), it is assumed that semantic segmentation of the image frame has been performed after step 204 of method (200). Semantic segmentation may identify different pixels of the image frame as belonging to a specific category, such as "pedestrian" or "vehicle".

[0062] In step 302, a segmentation binary mask is generated. The segmentation binary mask may directly reflect the result of the semantic segmentation performed in step 204 of method (200) after binarization. Separate segmentation masks may be generated for different objects. An example of a segmentation binary mask for the object "Pedestrian" is shown in Figure 4.

[0063] In step 304, segmentation boundaries are detected. Boundary pixel detection may be performed using an image processing and computer vision library such as "Skimage," or any other type of boundary detection algorithm. Segmentation boundary detection may be performed for each object identified by semantic segmentation. Segmentation boundary detection may be performed on the segmentation binary mask.

[0064] In step 306, the separated blobs of the object instances identified by semantic segmentation are merged into a single blob based on the boundary points obtained in step 304. Morphological operations such as a combination of expansion and contraction may be used. The output may be a merged blob for each instance of the object identified by semantic segmentation. An example shown in Figure 4 is provided, as will be discussed further below. In addition, different objects may be combined to form new objects. For example, the objects "road", "lane", and "curb" may be combined to form a new object "ground" that forms a horizontal plane. The availability of the object "ground" may be beneficial or essential for the placement of other objects, such as expansion content.

[0065] In step 308, islands representing instances of the object are detected and isolated. This operation may be performed using a segmentation binary mask. An example of the object type "pedestrian" is provided in Figure 4. In the example, there are four instances of the object "pedestrian". The operation in step 308 ensures that the four pedestrians are treated individually (e.g., tracked) rather than as a single object representing all four pedestrians together. In other words, the example in Figure 4 relates to an object of type "pedestrian", and the operation in step 308 ensures that all four instances of the object "pedestrian" are accessible. The operation in step 308 may also be enabled by a dictionary that may have been generated based on the output of step 204, in which different instances of the object may have been detected and labeled.

[0066] In step 310, a contiguous pixel boundary is generated for the object instance. Initially, the object instance may have been represented by an unordered list of boundary pixels, but the operation in step 310 yields an ordered list. The ordered list may be generated using any method, such as a shortest path algorithm or a shortest-minimum-cost path algorithm.

[0067] In an example of a pedestrian object, the starting point for the list of boundary pixels is the top of the head. Therefore, the list of boundary pixels completes the contour without a zigzag pattern by having points sequentially starting from the top of the head, then moving to the left shoulder, followed by more pixels on the left side, down to the left leg, then the left foot, then the right foot, moving up the right leg, down the right side, and ending with the last pixel adjacent to the starting pixel. The resulting list of boundary pixels may be used as input to a rendering engine.

[0068] Generating continuous pixel boundaries is computationally intensive, and different methods can provide different frame rates on the same computing platform. For example, a graph-based version (using the NetworkX library) is 6 fps, a distance vector-based version (using the Numpy library) is 27 fps, and a distance matrix-based version (using the Scipy library) is 32 fps.

[0069] If additional types of objects exist in step 312, the steps described above may be repeated until the steps of method (300) have been completed for all types of objects.

[0070] Figure 4 shows an example of merging multiple blobs associated with an instance (400) of an object into a single blob according to an embodiment of the present disclosure. An image from which semantic segmentation has been performed is initially available. The segmentation binary mask is obtained by binarizing the image frame identified by arrow (1). In the example, the segmentation binary mask is for an object type "pedestrian". In the example, there are four instances of the object "pedestrian".

[0071] The resulting representation of an instance in a segmentation binary mask may be suboptimal, as a single object is segmented into smaller, isolated chunks. In the example, one instance of an object represents the head of a pedestrian, not connected to the pedestrian's torso. In other words, the representation contains multiple isolated blobs. As indicated by the arrows (2a, 2b, 2c, 2d), the smaller blobs may be combined into larger blobs using morphological operations. Different results may be obtained depending on the parameterization of the morphological operations (such as dilation and sagging).

[0072] Figure 5 shows an example of a screenshot (500) according to an embodiment of the present disclosure. The screenshot illustrates the coexistence of real and virtual objects in a possible application. Real static objects include roads, buildings, etc. Real dynamic objects include cars and pedestrians. Virtual objects (cat and runner) are inserted in reasonable positions, i.e., positions that are in contact with the ground.

[0073] Figure 6 shows a computing system according to one or more embodiments. Embodiments may be implemented on a computer system. Figure 6 is a block diagram of a computer system (602) used to provide computational functions related to the algorithms, methods, functions, processes, flows, and procedures described herein, according to one implementation. The illustrated computer (602) is intended to encompass any computing device, such as a high-performance computing (HPC) device, a server, a desktop computer, a laptop / notebook computer, a wireless data port, a smartphone, a personal data assistant (PDA), a tablet computing device, one or more processors in these devices, or any other suitable processing device (including physical instances or virtual instances of computing devices, or both). Furthermore, the computer (602) may include input devices such as a keypad, keyboard, touchscreen, or other device capable of accepting user information, and output devices that transmit information related to the operation of the computer (602) (including digital data, visual information, or audio information, or a combination thereof, or a GUI).

[0074] Computer (602) may function as a client, network component, server, database or other persistence, or any other component (or combination thereof) of a computer system for performing the subject matter described in this disclosure. The illustrated computer (602) is communicatively coupled to a network (630). In some implementations, one or more components of computer (602) may be configured to operate in an environment including a cloud computing base, local, global, or other environment (or combination thereof).

[0075] At a high level, a computer (602) is an electronic computing device capable of receiving, transmitting, processing, storing, or managing data and information relating to the subject described. According to some implementations, a computer (602) may also include, or be communicably coupled with, application servers, email servers, web servers, caching servers, streaming data servers, business intelligence (BI) servers, or other servers (or combinations thereof).

[0076] Computer (602) can respond to received requests by receiving them via the network (630) from client applications (for example, running on another computer (602)) and processing the requests in appropriate software applications. Furthermore, requests may also be sent to computer (602) from internal users (for example, from a command console or by other appropriate means of access), external or third parties, other automated applications, and any other appropriate entities, individuals, systems, or computers.

[0077] Each component of the computer (602) can communicate using the system bus (603). In some implementations, any or all components of the computer (602) (hardware or software, or a combination of hardware and software) may interface with each other or with Interface (604) (or both) via the system bus (603) using an Application Programming Interface (API) (612) or a Service Layer (613) (or a combination of API (612) and Service Layer (613)). The API (612) may include specifications for routines, data structures, and object classes. The API (612) may be independent of or dependent on a computer language, and may refer to a complete interface, a single function, or even a set of APIs. The Service Layer (613) provides software services to the computer (602) or other components (whether illustrated or not) coupled to the computer (602) in a communicative manner. The functions of the computer (602) may be accessible to all service consumers using this Service Layer. Software services, such as those provided by the service layer (613), provide reusable and defined business functions through a defined interface. For example, the interface may be software written in Java, C++, or another suitable language that provides data in Extended Markup Language (XML) format or another suitable format. Although illustrated as an integrated component of Computer (602), alternative implementations may show the API (612) or service layer (613) as a separate component in relation to other components of Computer (602), or other components (whether illustrated or not) that are communicatively coupled to Computer (602).Furthermore, any or all parts of the API(612) or service layer(613) may be implemented as a child module or submodule of another software module, enterprise application, or hardware module without departing from the scope of this disclosure.

[0078] The computer (602) includes an interface (604). Although shown as a single interface (604) in Figure 6, two or more interfaces (604) may be used depending on the specific needs, requirements, or implementation of the computer (602). The interface (604) is used by the computer (602) to communicate with other systems in a distributed environment connected to the network (630). Generally, the interface (604) includes logic encoded in software or hardware, or a combination of software and hardware, and is operable to communicate with the network (630). More specifically, the interface (604) may include software that supports one or more communication protocols, thereby enabling the network (630) or the interface hardware to communicate internal and external physical signals of the illustrated computer (602).

[0079] The computer (602) includes at least one computer processor (605). Although shown as a single computer processor (605) in Figure 6, two or more processors may be used depending on the specific needs, requirements, or implementation of the computer (602). Generally, the computer processor (605) executes instructions and manipulates data in order to operate the computer (602) and to perform any algorithms, methods, functions, processes, flows, and procedures described herein.

[0080] The computer (602) also includes memory (606) that holds data for the computer (602) or for other components (or both) that may be connected to the network (630). For example, memory (606) may be a database that stores data consistent with this disclosure. Although shown as a single memory (606) in Figure 6, two or more memories may be used depending on the specific needs, requirements, or implementation of the computer (602) and the functions described. Although memory (606) is shown as an integral component of the computer (602), in alternative implementations, memory (606) may be external to the computer (602).

[0081] An application (607) is an algorithmic software engine that provides functionality to the specific needs, requirements, or implementation of the computer (602), particularly with respect to the functionality described herein. For example, an application (607) may function as one or more components, modules, applications, etc. Furthermore, although illustrated as a single application (607), an application (607) may be implemented as multiple applications (607) on the computer (602). In addition, although illustrated as being integrated with the computer (602), in alternative implementations, an application (607) may be external to the computer (602).

[0082] There may be any number of computers (602) including or outside of computer (602), each computer (602) communicating via a network (630). Furthermore, the terms “client,” “user,” and other appropriate terms may be used interchangeably as appropriate without departing from the scope of this disclosure. Furthermore, this disclosure is intended to show that many users may use one computer (602), or that one user may use multiple computers (602).

[0083] In some embodiments, the computer (602) is implemented as part of a cloud computing system. For example, the cloud computing system may include one or more remote servers along with various other cloud components such as cloud storage units and edge servers. In particular, the cloud computing system may perform one or more computing operations without direct active management by user devices or local computer systems. Thus, the cloud computing system may have different functions distributed across multiple locations from a central server, and it may be run using one or more internet connections. More specifically, the cloud computing system may operate according to one or more service models such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), Mobile “Backend” as a Service (MBaaS), Serverless Computing, Artificial Intelligence as a Service (AIaaS), and / or Function as a Service (FaaS).

[0084] Advantageously, the embodiments disclosed herein enable real-time, current-environment-based VR / AR experiences while moving within a vehicle without the requirement of camera movement to generate scenes that depict or utilize aspects of the environment surrounding the vehicle. Furthermore, the solutions disclosed herein do not use SLAM (Simultaneous Localization and Mapping) techniques and do not require actual 3D reconstruction (neither meshes nor voxels). This allows for less computational overhead and lower storage requirements for rendering VR / AR experiences to people inside (or possibly outside) the vehicle.

[0085] Although only a few exemplary embodiments have been described in detail above, those skilled in the art will readily understand that many modifications are possible in the exemplary embodiments without substantially departing from the present invention. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the following claims.

Claims

1. A method for augmented reality in automotive applications, By acquiring image frames from the car's camera, Identify at least one environmental object within the aforementioned image frame, Based on the aforementioned at least one environment object, the extended content is determined, Based on the aforementioned image frame and the aforementioned extended content, the extended image frame is rendered. The extended image frame is displayed to the user. method.

2. The method according to claim 1, wherein the image frame includes an image of the external environment of the automobile.

3. The method according to claim 1, wherein acquiring the image frame includes preprocessing specific to the camera.

4. The method according to claim 3, wherein the camera is a fisheye camera, and the preprocessing includes cropping the image frames acquired from the fisheye camera.

5. The method according to claim 1, wherein the at least one environmental object identified within the image frame includes at least one selected from the group consisting of another automobile, bicycle, motorcycle, road, lane markings, curb, traffic sign, building, animal, and pedestrian.

6. Identifying the at least one environmental object within the image frame is: Processing the image frames using machine learning-based image processing pre-trained with traffic environment image data, The method according to claim 1, including the method described in claim 1.

7. The method according to claim 6, wherein the machine learning-based image processing includes semantic segmentation of the image frame.

8. Identifying the at least one environmental object within the image frame further means, The step of generating at least one segmentation binary mask for the at least one environment object is included. The method according to claim 6.

9. Identifying the at least one environmental object within the image frame further means, This includes integrating the isolated blobs associated with one instance of the at least one environment object, The method according to claim 6.

10. Identifying the at least one environmental object within the image frame further means, With respect to one or more instances of the at least one environment object, the process includes separating the multiple instances. The method according to claim 6.

11. Identifying the at least one environmental object within the image frame further means, This includes generating a continuous pixel boundary for one instance of the at least one environment object. The method according to claim 6.

12. Determining the aforementioned extended content means To render the aforementioned extended image frame, the process includes selecting at least one of an avatar, a symbol, a label, and an overlay. The method according to claim 1.

13. The method according to claim 12, wherein the extended content is controllable from a user input device.

14. A system for augmented reality in automotive applications, An environment interpretation engine that identifies at least one environment object in an image frame acquired from a car camera, A content extension engine that determines extension content based on at least one environment object, A rendering engine that renders an extended image frame and displays the extended image to the user based on the aforementioned image frame and the extended content. A system that includes these features.

15. The system according to claim 14, further comprising a developer kit that provides an interface for facilitating the development of applications to be run on the system.

16. The system according to claim 15, wherein the application to be executed on the system can be selected by the user from among the plurality of applications.

17. The system according to claim 15, wherein the plurality of applications include at least one selected from game applications, educational applications, and entertainment applications.

18. The system further comprises a display selected from the group consisting of displays for portable user devices and displays for the automobile, The display displays the extended image frame. The system according to claim 14.

19. The system according to claim 14, further comprising a user input device for the user to control the extended content.

20. A non-temporary computer-readable medium (CRM) for storing computer-readable program code for augmented reality in automotive applications, wherein the computer-readable program code is stored in a computer system. By acquiring image frames from the car's camera, Identify at least one environmental object within the aforementioned image frame, Based on the aforementioned at least one environment object, the extended content is determined, Based on the aforementioned image frame and the aforementioned extended content, render the extended image frame. The extended image frame is displayed to the user. A non-temporary, computer-readable medium that enables an action to be performed.

Images