Image synthesis in multi-view automotive and robotic systems

Abstract

In various examples, two or more cameras in an automotive surround view system generate two or more input images to stitch or combine into a single stitched image. In one embodiment, to improve the quality of the stitched image, a feedback module computes two or more scores representing errors between the stitched image and one or more input images. If the computed scores indicate structural errors in the stitched image, the feedback module computes and applies one or more geometric transformations to apply to the one or more input images. If the computed scores indicate color errors in the stitched image, the feedback module computes and applies one or more photometric transformations to apply to the one or more input images.

Description

Background Technology

[0001] Automotive systems are increasingly employing visualization solutions to assist drivers, provide information, and enhance safety. However, the various technologies used to provide visualization in automotive applications all have their drawbacks. For example, cameras typically cannot capture all areas where safety issues might occur, or all areas that a viewer might be interested in. Attempting to provide a wider field of view due to inherent camera limitations or characteristics (such as position, orientation, and intrinsic parameters) can result in image distortion and / or artifacts. Any distorted visualization or artifacts in automotive visualization increase the difficulty for drivers, other vehicle users, safety monitors, and traditional image processing techniques to process the information provided by the automotive visualization system, as these artifacts and / or distorted visualizations do not reflect the real world. These distortions and / or artifacts can reduce driver safety by providing inaccurate or incomplete information. Summary of the Invention

[0002] Embodiments of this disclosure relate to improving image synthesis in automotive or robotic systems using quality assessment feedback. Systems and methods are disclosed for building improved panoramic images generated in automotive, robotic, or platform systems to enhance user safety and diagnostic capabilities. Image synthesis involves combining image data from two or more sources (e.g., cameras) for display in a single visualization engine. That is, capturing image data from each individual camera is combined or stitched together to present a panoramic view. Conventional synthesis processes often result in incorrect alignment between individual images, producing artifacts and anomalously blurred overall landscape views, which is particularly problematic in vehicle applications because artifacts can appear in safety-critical areas.

[0003] Traditional automotive visualization systems are open-loop, limited to pairwise combinations, and may utilize simple image combination techniques, such as homography estimation, which requires estimating the correspondence between identified feature points between image data from two separate images. However, since feature points only appear when distinct objects are present in the scene, this approach is ineffective and introduces geometric distortion in some scenarios where stitched images contain misaligned edges between overlapping portions. Furthermore, homography estimation introduces geometric distortion in scenes with depth.

[0004] Compared to traditional systems (such as those described above), the improved image synthesis in automotive systems utilizes quality assessment feedback in a closed-loop synthesis quality optimization module to guide various adjustments for each image, thereby minimizing artifacts and other distortions. The synthesis quality optimization module receives calibration parameters and image data as input from two or more cameras, and may also optionally receive data from other vehicle sensors. The module calculates the overlap region for each image pair and, based on this overlap region, calculates one or more synthesis quality scores. If any synthesis quality score falls below a threshold, the module calculates and applies one or more geometric or photometric transformations to perform on a subset of the input image data. Based on the transformed image data, the module loops until one or more synthesis quality scores are all above or equal to the threshold. During each loop, the module calculates one or more synthesis quality scores and applies one or more geometric or photometric transformations. Attached Figure Description

[0005] The system and method for improving image synthesis in automotive systems using quality assessment feedback are described in detail below with reference to the accompanying drawings, wherein:

[0006] Figure 1 This is a block diagram illustrating a synthesis pipeline according to some embodiments of the present disclosure;

[0007] Figure 2A This is an illustration of a composite image using two images according to some embodiments of this disclosure;

[0008] Figure 2B This is an illustration synthesized from bowl projection images before and after optimization according to some embodiments;

[0009] Figure 3 This is a block diagram illustrating a synthesis engine pipeline according to some embodiments of the present disclosure;

[0010] Figure 4 This is a block diagram illustrating a synthesis module for performing improved image synthesis according to some embodiments of the present disclosure;

[0011] Figure 5 This is a block diagram illustrating a synthesis quality optimization module according to some embodiments of the present disclosure;

[0012] Figure 6 This is a block diagram illustrating a quality assessment module according to some embodiments of the present disclosure;

[0013] Figure 7 The process of performing improved synthesis using synthesis quality optimization according to some embodiments of this disclosure is illustrated;

[0014] Figure 8AThese are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0015] Figure 8B According to some embodiments of this disclosure Figure 8A Examples of camera positions and fields of view for autonomous vehicles;

[0016] Figure 8C According to some embodiments of this disclosure Figure 8A A block diagram of an example system architecture for an example autonomous vehicle;

[0017] Figure 8D Cloud-based servers and according to some embodiments of this disclosure Figure 8A A system diagram illustrating communication between autonomous vehicles;

[0018] Figure 9 This is a block diagram of an example computing device applicable to implementing some embodiments of the present disclosure. Detailed Implementation

[0019] Systems and methods related to an improved image synthesis pipeline are disclosed, which utilize quality assessment feedback used by automotive systems and platforms or robotic systems and platforms with surround-view capabilities.

[0020] In multi-view systems, such as those used in surround-view vehicle systems, individual cameras capture images corresponding to a portion of the landscape. Image data captured from each individual camera is combined or stitched together to present a complete view of the landscape in a single image through the synthesis of two or more images. Synthesis is performed using a compositing engine or compositing pipeline as a component of the surround-view vehicle system or any other system in which multiple overlapping images must be combined.

[0021] When a surround-view vehicle system, as further described herein, stitches together two or more images captured by onboard cameras, the resulting one or more images are often misaligned, containing artifacts and anomalies that obscure or disrupt the overall landscape view. Examples of this will be combined below. Figure 2A Further description. Traditional methods for synthesizing images focus on homography estimation to align unstitched images.

[0022] Using homography estimation, traditional open-loop surround-view car systems combine overlapping images by estimating the correspondences between feature points identified in individual image data. That is, a system implementing homography estimation for image synthesis identifies feature points in each of the two images to be combined and estimates how these feature points correspond to each other. Using this correspondence, overlapping regions are identified and the images are combined. However, since feature points only appear when different objects are present in the scene, traditional methods (such as homography estimation) are ineffective for some scenes and introduce geometric distortion, which is evident when stitching images containing misaligned edges between overlapping portions. Furthermore, homography estimation introduces geometric distortion for the entire landscape view in scenes with depth.

[0023] Traditional methods also have limited applicability, only applicable to paired combinations. These traditional methods cannot be used in multi-view systems using more than two cameras. In addition to the limitations of homography estimation, multi-view systems introduce photometric distortion, where each camera has different exposure levels, resulting in inconsistent color values ​​across the stitched landscape view, as shown below. Figure 2A Further description.

[0024] In contrast, closed-loop feedback-based methods for image synthesis in automotive systems and platforms or robotic systems and platforms with surround-view capabilities can be used to synthesize two or more images in settings where homography estimation is unavailable or introduces significant artifacts. Furthermore, feedback-based methods are robust to photometric distortion and can perform color correction to match images used as input to automotive systems and platforms, or robotic systems and platforms with surround-view capabilities, enabling the implementation of the feedback-based methods.

[0025] A feedback-based image compositing method for automotive systems and platforms, or robotic systems and platforms with surround-view capabilities, and other systems requiring image compositing, utilizes a closed-loop compositing quality optimization module to guide geometric adjustments and photometric alignment between two or more stitched images. This compositing quality optimization module is combined as follows: Figure 4-6 As described, data is received as input image data and intrinsic and extrinsic calibration parameters from two or more cameras in an automotive system and platform or a robotic system and platform with surround-view capabilities, and data may be selectively received from other vehicle sensors, as further described herein.

[0026] Each image in a pair is first transformed from fisheye space to synthetic space, as shown in the combination in this paper. Figure 8B Each camera captures a further description. Combine the engine or module as follows: Figure 1 and Figure 4As described, two or more individual images in the synthesis space are then stitched together by projecting each image onto the image coordinates of a virtual camera, where the virtual camera represents the stitched or combined image pair including an overlapping region. This overlapping region is calculated for each image pair and indicates the portion of each image from each camera that overlaps in the image space of the virtual camera. Based on this overlapping region, the synthesis quality optimization module performs one or more quality evaluation operations to calculate one or more synthesis quality scores and determine one or more transformations to be applied to the respective input images of each image pair.

[0027] For example, in one embodiment, the synthesis quality optimization module performs one or more quality assessment operations to calculate the geometric and photometric scores for each stitched image, as combined below. Figure 6 Further described. The geometric score is a numerical value representing the amount or degree of structural error resulting from the compositing engine synthesizing two or more images into a single image, as combined below. Figure 5 and Figure 6 Further described. In one embodiment, a higher geometric score indicates that the stitched image contains fewer structural errors and is closer to representing an ideal combination of the two images by the compositing engine. A lower geometric score indicates that the stitched image contains more structural errors, representing improper compositing by the compositing engine. The photometric score is a numerical value representing the degree of color difference between the two images to be stitched, as shown below in conjunction with Figure 2 and Figure 6 Further described. In one embodiment, a higher score indicates that each image to be stitched is very similar to the other images in terms of color attributes. A lower score indicates that each image to be stitched includes significant differences in color attributes.

[0028] A synthesis quality optimization module, as follows: Figure 5 Further described, the geometry and photometric scores are analyzed. In one embodiment, if the geometry score is below a threshold (indicating poor synthesis quality), the synthesis quality optimization module applies geometry alignment. During geometry alignment, the synthesis quality optimization module calculates a 3D transformation, as described below. Figure 5 Further described, this is applied to one or more stitched images. In one embodiment, the 3D transformation is a geometric transformation that modifies one or more stitched images and adjusts the external calibration of one camera while keeping the external calibration of another camera unchanged, thereby maximizing the geometric quality matrix. Once the compositing quality optimization module has calculated the 3D transformation, it applies the transformation to generate stitched or virtual camera images with the highest possible alignment.

[0029] In one embodiment, if the photometric score is below a threshold (indicating poor color matching between two input images), the synthesis quality optimization module applies photometric adjustments, as follows: Figure 5Further described. During photometric adjustment, the synthesis quality optimization module improves photometric quality by matching the color intensity and color angle between input images from two cameras. The synthesis quality optimization module calculates and applies a transform to match the intensity and angle of the target input image to the source input image of a single RGB channel, where the transform is based on applying scaling and offsets to the color channels of the image. To find the optimal scaling and offset, the synthesis quality optimization module directly searches for values ​​to maximize the photometric score, thereby maximizing the synthesis quality. To improve performance and reduce the time required to determine the optimal scaling and offset, in one embodiment, the synthesis quality optimization module reduces the search space by calculating the initial scaling and offset as the average of both the source and target images.

[0030] Because automotive systems require real-time performance, in one embodiment, the synthetic quality optimization module utilizes optical flow vectors to reduce the search space required to compute 3D transformations when performing geometric adjustments, as described below. Figure 5 Further described. The reduced search space facilitates real-time determination of 3D transformations. To reduce the search space, optical flow vectors help the synthesis quality optimization module identify the orientation of the rotated image to improve the geometric quality score, thereby eliminating any transformations that do not rotate the image in that particular orientation.

[0031] While the synthesis of two images from two cameras in an automotive surround-view system is widely used for illustrative purposes, it should be noted that the techniques described herein can be adapted for other uses. For example, in one embodiment, improved image synthesis using the various techniques described herein can be used to facilitate improved virtual reality systems. In another embodiment, improved image synthesis using the various techniques described herein can be used to facilitate autonomous or semi-autonomous vehicles or other automotive applications, where stitched images are input into neural networks or other machine learning techniques used by one of the automotive systems (e.g., control systems, emergency braking). In another embodiment, improved image synthesis using the various techniques described herein can be used to improve safety camera systems. In yet another embodiment, improved image synthesis using the various techniques described herein can be used for medical applications. For example, improved image synthesis can be used to stitch together images and / or videos from multiple orthodontic cameras to provide a medical specialist or robotic surgeon with an improved view of the entire area surrounding an organ being operated on. In yet another embodiment, improved image synthesis using the various techniques described herein can be used to align projectors in cinemas or other visualization systems that require combining views from multiple cameras (e.g., IMAX). In another embodiment, the construction equipment may employ image synthesis improved using various available techniques described herein, for example by showing the excavator operator the area being excavated, where the view may be obstructed by the bucket or other construction equipment components.

[0032] In one embodiment, the compositing techniques described herein can be used to facilitate the generation of two or more images to be combined or stitched in an automotive surround-view system including two or more cameras. In one embodiment, in an automotive surround-view system including three or more cameras generating images to be stitched, the improved image compositing pipeline described herein, utilizing quality assessment feedback, is applied in two different ways. First, in one embodiment, a compositing quality optimization module can continuously evaluate and improve the quality for each overlapping region between three or more images to be combined or stitched, captured from three or more cameras. The adjusted image is used as a reference image when adjusting the next overlapping region. Second, in one embodiment, the compositing quality optimization module optimizes the compositing quality for all overlapping regions between three or more images to be combined or stitched, captured from three or more cameras. To optimize geometric quality, the compositing quality optimization module aggregates the geometric scores of all overlapping regions between the images to form a global geometric score for the stitched image as a whole. Based on this score, the compositing quality optimization module performs a search in the latent transform space of each image, as combined below. Figure 5 The description focuses on optimizing the overall geometric score. To optimize photometric quality, the synthesis quality optimization module aggregates the photometric scores from all overlapping regions between three or more input images to form the overall photometric score of the stitched image. Based on this score, the synthesis quality optimization module searches the scale and offset space of each input image to maximize the overall photometric score.

[0033] Various techniques have been described in the preceding and following sections. For illustrative purposes, specific configurations and details are elaborated to provide a thorough understanding of the possible ways to implement these techniques. However, it will be apparent that the techniques described below can be practiced in different configurations without specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the techniques described.

[0034] refer to Figure 1 , Figure 1 These are example architectures for performing video and / or image composition according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are presented by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0035] Figure 1This is a block diagram illustrating a compositing pipeline or compositing block according to some embodiments of the present disclosure. In the compositing pipeline, compositing engine 110 receives multi-view input 104. In one embodiment, the multi-view input is data, such as an image from camera 102, data from sensor 106, and / or calibration parameters 108. In one embodiment, the multi-view input 104 includes camera 102, such as surround camera 874, wide-angle camera 870, mid-range camera 898, long-range camera 898, or stereo camera 868, as combined below. Figure 8B Further described herein. In one embodiment, camera 102 is any hardware device that captures image and / or video data. For example, in one embodiment, camera 102 is a fisheye camera, or includes a fisheye lens, and generates images with fisheye projection or perspective. In another embodiment, camera 102 is any other type of camera used to facilitate visualization 118 in a vehicle surround-view system. In yet another embodiment, camera 102 is any type of camera used to facilitate visualization in a system performing image compositing, as further described herein.

[0036] Two or more cameras 102 generate images for a multi-view input 104. In one embodiment, the image is data generated by one or more cameras 102 representing a location within the field of view, such as the area around a vehicle in an automotive surround-view system. In one embodiment, the multi-view input 104 includes calibration parameters 108. In one embodiment, calibration parameters 108 are a set of data values ​​that can be used to configure one or more cameras 102. For example, in one embodiment, calibration parameters 108 include color, contrast, and brightness. In another embodiment, calibration parameters 108 are a set of data values ​​associated with other components of the vehicle or image capture system, such as sensor 106. In one embodiment, the sensor is a software and / or hardware component for collecting environment-related data, such as a RADAR sensor 860 or a LIDAR sensor 864, as further described herein. Calibration parameters 108 include intrinsic and extrinsic parameters. In one embodiment, intrinsic parameters are focal length, optical center or principal point, skew coefficient, and / or distortion parameters. In one embodiment, extrinsic parameters are camera rotation and translation. For example, in one embodiment, the external calibration parameters 108 include positional data representing rotation and other geometric adjustments to one or more images from one or more cameras 102. In one embodiment, the multi-view input includes two or more images from two or more cameras 102 and calibration parameters 108 associated with the two or more images. In one embodiment, each image is associated with its own separate set of calibration parameters 108. In another embodiment, the two or more cameras 102 share one or more sets of calibration parameters 108. In one embodiment, the calibration parameters 108 may be unavailable. If the calibration parameters 108 are unavailable, feature matching techniques such as homography estimation are used to estimate a transformation that can be used to transform the two or more multi-view input 104 images into a common stitching space or a single image.

[0037] Compositing engine 110 receives two or more images from camera 102 and associated calibration parameters 108, along with optional sensor 106 data, as multi-view input 104. In one embodiment, compositing engine 110 is a combination of data values ​​and software instructions that, when executed, combine and align two or more images from two or more cameras 102 into a single image with a common coordinate system. In one embodiment, compositing engine 110 receives multi-view input 104 from an ingestion block comprising two or more cameras 102. In another embodiment, compositing engine 110 receives multi-view input 104 from any grouping or implementation of two or more cameras 102.

[0038] The compositing engine uses multi-view input 104, including images from camera 102, optional sensor 106 data, and calibration parameters 108, to generate output 112, which includes one or more stitched images 114 and updated calibration parameters 116. The one or more stitched images 114 are image data generated by the compositing engine 110 based at least in part on two or more input images from camera 102. In one embodiment, the stitched image 114 is a 2D image or texture comprising any data format supported by an automotive surround-view system further described herein in one embodiment. In another embodiment, the stitched image 114 is a 3D image or texture.

[0039] Compositing engine 110 combines or stitches two or more multi-view input 104 images by aligning each multi-view input 104 image as a single image or a stitched image 114 to a common coordinate system, as shown below. Figure 4 Further described. In one embodiment, compositing engine 110 aligns two or more multiview input 104 images into one or more stitched images 114 by aligning the images based on camera calibration parameters 108. In another embodiment, compositing engine 110 aligns two or more multiview input 104 images into one or more stitched images 114 by identifying one or more features or landmarks and adjusting calibration parameters 108 to align each of the two or more multiview input 104 images. In yet another embodiment, compositing engine 110 aligns two or more multiview input 104 images at least in part based on sensor 106 data. In yet another embodiment, compositing engine 110 utilizes any other technique to align each multiview input 104 image into the output 112 stitched image 114.

[0040] In one embodiment, adjustments made to the multi-view input 104 image result in updated calibration parameters 116. The updated calibration parameters 116, output 112 by the compositing engine 110, include data representing new calibration values ​​that can be used by two or more cameras 102 to further capture image and / or video data, or to adjust additional components, such as sensor 106. In one embodiment, the updated calibration parameters 116 include adjustments to calibration data corresponding to structural or geometric adjustments, such as rotation and translation. In another embodiment, the updated calibration parameters 116 include adjustments to calibration data corresponding to color or photometric adjustments, such as brightness and / or contrast. In one embodiment, the updated calibration parameters 116 include both geometric and photometric adjustments.

[0041] The compositing engine 110 determines the quality of one or more stitched images, as follows: Figure 5 and 6Further described. Based on quality analysis, the compositing engine determines one or more transformations to apply to the multi-view input 104 images in order to generate an optimal output 112 stitched image 114. In one embodiment, a single-pass analysis determines the photometric and geometric quality of the stitched image 114 generated by the compositing engine 110. In another embodiment, the compositing engine 110 performs multiple passes using a feedback mechanism to improve the quality of the output 112 stitched image 114, as described below. Figure 5 Any adjustments made to the multi-view input 104 image are reflected by the updated calibration parameter 116.

[0042] In one embodiment, the output 112, including one or more stitched images 114 and any updated calibration parameters 116, is utilized by a visualization 118 module, such as a visualization engine or block in a vehicle surround view system. In another embodiment, the output 112 from the compositing engine 110 is used by any other visualization 118 system to display or otherwise utilize one or more stitched images 114. In one embodiment, one or more additional transformations are applied by the compositing pipeline prior to visualization 118, as combined below. Figure 3 Further description.

[0043] Figure 2A This is an illustration of the synthesis of two images according to some embodiments of the present disclosure. During synthesis, two or more cameras capture a left image 202 and a right image 204. In one embodiment, the left image 202 is data including a representation of visual information corresponding to a left position within the scene, and the right image 204 is data including a representation of visual information corresponding to a right position within the same scene. In automotive surround-view camera systems or other multi-view systems, as described above... Figure 1 As described herein and further described, left image 202 and right image 204 include an overlapping region 206. In one embodiment, the overlapping region 206 is data representing a shared projection space within a scene captured by two or more cameras. In one embodiment, the overlapping region 206 includes one or more landmarks or features that can be used to align left image 202 and right image 204. In one embodiment, both left image 202 and right image 204 include one or more landmarks or features in the overlapping region 206.

[0044] A synthesis engine, such as the one described above. Figure 1 Description and the following text combined Figure 4-6Further described, a stitched image 208 is generated at least in part based on the left image 202 and the right image 204. In one embodiment, the stitched image 208 includes a stitching region 210 containing projection information about a scene from the overlapping region 206 of the left image 202 and the right image 204. The stitching region 210 is data representing projections within a scene that includes the overlapping region 206 of the left image 202 combined with or aligned with the overlapping region 206 of the right image 204.

[0045] In one embodiment, the stitching region 216 of the stitched image 208 contains geometric errors, wherein elements from the left image 212 of the stitching region 216 are not correctly aligned with elements from the right image 214 of the stitching region 216. The geometric errors are quantified by the compositing engine using a geometric quality score, as described below. Figure 5 As described. In one embodiment, the stitching region 222 of the stitched image 208 includes a photometric error, wherein a portion from the left image 218 in the stitching region 222 includes different color levels compared to a portion from the right image 220 in the stitching region 222. The compositing engine uses a photometric quality fraction to quantify the different color levels, as described below. Figure 5 The compositing engine calculates or otherwise determines one or more transformations or other adjustments to be applied to the left image 202 and / or the right image 204, at least in part, based on geometric quality fraction and photometric quality fraction, as described below. Figure 5 and Figure 6 Further description.

[0046] Figure 2B This is an illustration of bowl projection image synthesis before 222 and after optimization 224, according to some embodiments. The bowl projection image synthesis before optimization 222 is a direct image synthesis, comprising several images already stitched together by a car surround view system. Before optimization, visual artifacts such as misaligned building edges in the upper right quadrant of the bowl projection image synthesis 222 are visible. The bowl projection image synthesis after optimization 224 is a mixed and / or stitched image that corrects geometric and photometric distortions using various techniques further described herein. After optimization 224, the visual artifacts visible in the bowl projection image synthesis before optimization 222 are reduced or become unobservable.

[0047] Figure 3 This is a block diagram illustrating a synthesis engine 302 pipeline according to some embodiments of the present disclosure. The synthesis engine 302 pipeline, as... Figure 3The diagram illustrates a series of software and / or hardware modules that, when executed, combine the multi-view input 304 images into a single image comprising elements of the multi-view input images 304 to generate a scene. In one embodiment, the compositing engine 302 pipeline is a component of an automotive surround-view system, as further described herein. In another embodiment, the compositing engine is a component of any other visualization system, such as virtual reality, or any other application that requires combining the multi-view inputs 304 into a single output 314 for visualization.

[0048] In one embodiment, the compositing engine 302 receives multi-view input 304 as input from a data ingestion block comprising two or more cameras and calibration parameters, as described above. Figure 1 As described herein and further described herein. In another embodiment, the compositing engine 302 receives any other data as input that can be used to facilitate image compositing.

[0049] In one embodiment, the compositing engine 302 includes a dewarp 306 module or a fisheye correction module. In one embodiment, the dewarp 306 module comprises data values ​​and software instructions that, when executed, transform the multi-view input 304 image from image space to compositing space. In one embodiment, image space is a fisheye projection of image data, or image data including fisheye coordinates. In one embodiment, compositing space is an equidistant rectangular or equidistant rectangular view projection space. In another embodiment, compositing space is a linear projection space. In one embodiment, compositing space is any projection space that the compositing engine 302 can use to project, combine, or otherwise utilize the multi-view input 302 image to generate an output 314 usable for visualization.

[0050] In one embodiment, the dewarping 306 module uses forward mapping to project the multi-view input 304 from a 2D image space to a 3D compositing space or other projection space. In forward mapping, the dewarping 306 module scans each multi-view input 304 image pixel by pixel and copies them to the appropriate positions in the compositing space image. In another embodiment, the dewarping 306 module uses reverse mapping to project the multi-view input 304 from a 2D image space to a 3D compositing space or other projection space. In reverse mapping, the dewarping 306 module iterates through each pixel of the target compositing space image and acquires or samples the correct pixels from one or more multi-view input 304 images. A single point in an image from one input camera is mapped to an epipolar line in another input camera. Each multi-view input 304 image is projected from a 2D input space to a 3D compositing space, where the depth of each pixel can be adjusted based on one or more transformations, as combined below. Figure 5In one embodiment, each 2D multiview input 304 image is mapped to an initial depth 0 in the region closer to the multiview camera sensor, as further described herein. In another embodiment, if the translation between the multiview input 304 images is small or negligible, or the scene depth is far, each 2D multiview input 304 image is mapped to an initial infinite depth. For example, if the images captured by each multiview input 304 camera do not contain overlapping or represent distant scenes, no depth adjustment will result in overlapping areas of the multiview input 304 images, and the initial depth will be mapped as infinite. In another embodiment, each 2D multiview input 304 image is mapped to any depth necessary to represent the perspective associated with the multiview input 304 image. In another embodiment, the dewarping 306 module maps each 2D multiview input 304 image to any depth estimated by any sensor or additional method, a LiDAR-based depth estimate, a monocular camera-based depth estimate, or a multiview camera-based depth estimate. In one embodiment, the dewarping module 306 maps each 2D multi-view input 304 image to any depth estimated using any technique to estimate the depth available in the surround-view system. In one embodiment, the dewarping module 306 uses one or more calibration parameters received by the compositing engine 302 as the multi-view input 304. In another embodiment, the dewarping module 306 does not use the calibration parameters received by the compositing engine 302 as the multi-view input 304.

[0051] In one embodiment, the compositing engine 302 includes a compositing module 308. In one embodiment, the compositing module 308 is a set of data values ​​and software instructions that, upon execution, combine and align input images (e.g., camera frames received as part of a multi-view input) into one or more output images 314 having a common coordinate system. In one embodiment, the compositing module 308 performs the alignment of two or more multi-view input images 304 (e.g., video frames or camera frames) based on one or more multi-view input 304 calibration parameters. In one embodiment, the compositing module 308 uses feature information from the two or more images to align the two or more multi-view input images 304. In another embodiment, the compositing module 308 uses parallax-based thinning to align the two or more multi-view input images 304. In yet another embodiment, the compositing module 308 detects structural artifacts or misalignments and refines the calibration parameters to adjust the stitched images. To detect structural artifacts or misalignments, the compositing module 308 combines the following... Figure 4 and 5 Further described, quality analysis and feedback are used to improve the quality of synthesis.

[0052] In one embodiment, the compositing 308 module combines two or more multi-view input 304 images during 2D-to-3D dewarping 306 using reverse mapping. That is, a stitched image representing the 3D virtual camera projection space receives information about the location of each 3D pixel from each of the two or more multi-view input 304 images. A mixing 310 module is used to blend the individual pixels corresponding to the multiple multi-view input 304 images together. In one embodiment, the mixing 310 module is a set of data values ​​and software instructions that, when executed, create a smooth pixel-level transition between the two or more multi-view input 304 images combined or projected onto a shared 3D projection space by the compositing 308 module. In one embodiment, the mixing 310 module combines pixel data from two or more multi-view input 304 images to generate output pixels. In one embodiment, the mixing 310 module combines single pixel data from the 3D stitching space into an output 314 2D image using forward or reverse mapping, as described above. In another embodiment, the mixing 310 module combines data from multiple pixels in each of the two or more multi-view input 304 images to generate a single or group of output pixels. In one embodiment, the mixing module 310 performs various mixing techniques, such as alpha or weighted mixing, multi-band mixing, gradient mixing, or optimal cutting. In another embodiment, the mixing module 310 performs any other mixing techniques to combine or otherwise mix pixel data from the multi-view input images to achieve a smooth transition between the multi-view input images in the stitched output image 314.

[0053] In one embodiment, the compositing engine 302 includes a projection 312 module. In one embodiment, the projection 312 module comprises data values ​​and software instructions that, when executed, project stitched image data from the compositing space as described above onto a fixed bowl or bowl-view projection space. In one embodiment, the projection 312 module generates an output 314 that can be used by a visualization system, as described above. Figure 1 As described. Output 314 includes data structures, such as image data and calibration parameters, which can be used to visualize a scene, such as the area around a vehicle in an automotive surround-view system. For example, in one embodiment, output 314 includes data structures consumed by an embedded visualization system or a non-embedded visualization block. In one embodiment, these output 314 data structures include 2D stitched images and / or textures. In another embodiment, the output 314 data structures include 3D stitched images and / or textures. In one embodiment, the output 314 data structure is a calibrated camera image. In one embodiment, the output 314 data structure includes data representing a bowl mesh or texture mesh mapping.

[0054] Synthetic Engine Pipeline 302, such as Figure 3As shown, it includes multiple software modules for performing image and / or video compositing. In one embodiment, the compositing engine 302 includes additional modules for facilitating image and / or video compositing specific to systems that integrate or otherwise utilize the compositing engine 302. For example, in one embodiment, the compositing engine 302 includes one or more reconstruction modules for reconstructing undercarriage texture data based on vehicle odometer data from an automotive system. In other systems, various software and / or hardware modules may be used to perform one or more additional steps to facilitate the operation of the compositing engine 302.

[0055] Figure 4 This is a block diagram illustrating a synthesis 402 module for performing improved image compositing according to some embodiments of the present disclosure. In one embodiment, the synthesis 402 module receives two or more 2D images received from two or more cameras as input 404. In another embodiment, the synthesis 402 module receives a 3D virtual camera image including combined or stitched image data as input 404, as a result of a de-distortion module performing a reverse mapping between 2D image data from two or more cameras and a 3D virtual camera or projection space.

[0056] In one embodiment, the compositing module 402 generates a stitched image 406 by executing software instructions that, upon execution, use a reverse mapping to convert two or more 2D input images into a 3D projection space or compositing space, wherein each pixel space in the 3D projection is derived from one or more 2D input images. In another embodiment, as described above... Figure 3 The dedistortion module generates a stitched image, and the synthesis module 402 only performs synthesis quality optimization 408.

[0057] The compositing module 402 includes a compositing quality optimization module 408. In one embodiment, the compositing quality optimization module 408 comprises data values ​​and software instructions that, upon execution, calculate one or more quality scores based at least in part on the 3D stitched image generated by the compositing module 402 406, as combined below. Figure 4 Further described. In another embodiment, the synthesis quality optimization module 408 calculates one or more quality scores, at least in part, based on the 3D stitched image generated by the de-distortion module and provided as input to the synthesis module 402, as described above. Figure 3 As described. In another embodiment, the synthesis quality optimization module 408 calculates one or more quality scores based on the 2D stitched image input 404 to the synthesis module 402 or generated by the synthesis module 402 406.

[0058] The synthesis quality optimization module 408 analyzes the 2D or 3D stitched image to generate one or more scores, and if any of the scores falls below a threshold, the synthesis quality optimization module 408 computes one or more transformations to apply to the input image 404 used to generate the 2D or 3D stitched image. In one embodiment, the threshold used for analyzing the scores computed by the synthesis quality optimization module is a constant value. In another embodiment, the threshold used for analyzing the scores computed by the synthesis quality optimization module is variable. In one embodiment, one or more neural network operations are used to infer an optimal or acceptable threshold to determine the threshold.

[0059] In one embodiment, the synthesis quality optimization 408 module provides feedback to any module responsible for generating the stitched image 406, such as those described above. In one embodiment, this feedback is one or more transforms to be applied to one or more 2D input images 404. In another embodiment, the synthesis quality optimization 408 module provides feedback on one or more updated calibration parameters available from two or more cameras combined with two or more 2D input images.

[0060] In one embodiment, the synthesis quality optimization module 408 outputs a 3D virtual camera projection or stitched image, which includes one or more transformations applied to one or more 2D input images 404. In another embodiment, the synthesis quality optimization module 408 uses a reverse mapping to project the 3D virtual camera projection or stitched image into 2D space and outputs the resulting 2D stitched image 416.

[0061] Figure 5 This is a block diagram illustrating a synthesis quality optimization module according to some embodiments of the present disclosure. In one embodiment, the synthesis quality optimization module 502 receives a 3D stitched image generated 506 using a dewarping module or a synthesis module as input 504, as described above. Figure 3 and Figure 4 As described. In another embodiment, the synthesis quality optimization module 502 receives the 2D stitched image as input 504. The synthesis quality optimization module 502 includes a quality evaluation block 508. In one embodiment, the quality evaluation block 508 is data values ​​and software instructions that, when executed, generate one or more scores as the result of one or more 2D or 3D input images.

[0062] In one embodiment, the quality assessment of 508 blocks generates a geometric quality score. The geometric quality score is combined as follows: Figure 6 Further described are data values ​​indicating the quality of how well two or more 2D input images have been stitched together into a 2D or 3D stitched image. That is, the geometric quality score. geometricIn one embodiment, it indicates whether the stitched image contains errors. The quality assessment block 508 calculates a geometric quality score by applying one or more mathematical analyses of the 2D and / or 3D input images 504, combining the 2D or 3D stitched image 506. geometric As shown below Figure 6 Further described. In one embodiment, the quality assessment 508 blocks dynamically calculate a geometric quality score based on the image content. geometric In one embodiment, the image content includes objects detected by an object detector, lane marking detector, traffic light detector, or any other object detection mechanism critical to safety and visual quality. For example, when misalignment occurs in human-sensitive areas (e.g., cars, pedestrians, lane markings, or any other visual safety indicator), the quality assessment block 508 reduces the geometric quality score. geometric .

[0063] In one embodiment, the quality assessment of 508 blocks generates a photometric quality score. photometric Photometric quality score photometric As shown below Figure 6 Further described are data values ​​indicating the degree of matching of each color space associated with each input image from two or more cameras in a 2D or 3D stitched 506 image. In one embodiment, the photometric quality score is... photometric Indicates whether the color and / or brightness and contrast levels between the input 504 images are significantly different, resulting in distorted 2D or 3D stitched images 506. In one embodiment, the quality evaluation block 508 calculates a photometric score based on color intensity score and color angle score. photometric Given two cameras, c1 and c2, the color intensity score is calculated as follows:

[0064]

[0065] Color angle fractions are calculated as follows:

[0066]

[0067] In one embodiment, the photometric score photometric Calculated as a score photometric =score intensity +score angle In another embodiment, a score is used. intensity and score angle Any other combination of these can be used to calculate the photometric score. For the geometric mass score... geometric and photometric quality score photometricIf the value is 1, the 2D or 3D stitched image is an ideal combination of two or more input 504 images. Any value less than 1 indicates a difference between the 2D or 3D stitched image and one or more input 504 images from cameras c1 and c2. A value significantly less than 1 indicates a significant difference between the 2D or 3D stitched image and the input 504 images from cameras c1 and c2.

[0068] In one embodiment, the quality assessment block 508 integrates the time consistency score into the synthetic quality score, such as the geometric score. geometric and / or photometric score photometric The Quality Assessment Block 508 generates a temporal consistency score by warping the stitched image according to time intervals or other stitched images within a window. In one embodiment, warping is a transformation in which one or more objects in the first stitched image are mapped to one or more locations of objects in the second stitched image within a time interval. In another embodiment, warping is a transformation in which the entire first stitched image is transformed to match the second stitched image within a time interval. As further described herein, the temporal transformation is performed at least in part based on one or more optical flow vectors. To compute the temporal consistency score, the Quality Assessment Block 508 compares the current stitched image with warped stitched images from neighboring frames within a certain time interval using any mathematical analysis, such as the Structural Similarity Index (SSIM) measure, which is combined below. Figure 6 The description refers to the color intensity and angle of the geometric fraction or photometric fraction of the integration time consistency. For illustrative purposes, in one embodiment, the quality assessment block 508 uses SSIM to compare the current stitched image with the distorted stitched image. In another embodiment, the quality assessment block 508 may use any other mathematical analysis techniques further described herein to compare the current stitched image with the distorted stitched image.

[0069] The synthesis quality optimization module 502 includes a quality score verification block 510. In one embodiment, the quality score verification block 510 is a set of data values ​​and software instructions that, when executed, compare one or more quality scores output by the quality assessment block 508 with one or more threshold numerical data values. In another embodiment, the quality score verification block 510 compares one or more quality scores output by the quality assessment block 508 with one or more numerical values ​​computed by a classifier using one or more machine learning techniques (e.g., neural networks). In one embodiment, one or more machine learning techniques learn a mapping between one or more objective synthesis quality scores and one or more subjective synthesis quality scores. In one embodiment, an objective quality score is a score reflecting the overall misalignment in a scene captured by one or more images. A subjective quality score is a score reflecting the misalignment between objects in a scene captured by one or more images. For example, in some cases, human vision may focus more on cars, pedestrians, lane markings, or other obstacles while driving and / or parking. In one embodiment, the automotive surround view system, utilizing the synthetic quality optimization module 502 to perform quality assessment 508, focuses on misalignments in these human visual focus areas and employs one or more machine learning techniques to map objective quality scores to lower subjective quality scores. The following is combined with... Figure 6 Further descriptions of various examples of objective and subjective scores.

[0070] If the quality score verification block 510 determines the geometric quality score calculated by the quality assessment block 508. geometric If the result is below a threshold, then the synthesis quality optimization 502 performs geometric alignment 512. In one embodiment, geometric alignment 512 is a combination of data values ​​and software instructions that, when executed, align unstitched input 504 images to improve the synthesis quality score. geometric In one embodiment, the technique used to perform alignment is any technique for improving synthetic alignment, such as pixel / feature / depth / parallax-based alignment, improved depth estimation, or any other technique used to perform alignment. In one embodiment, geometric alignment 512 calculates one or more transformations to adjust a portion of the 2D input 504 or 3D stitching 506 image. In another embodiment, geometric alignment 512 calculates one or more transformations to adjust calibration parameters associated with the source camera c1 and the target camera c2. In yet another embodiment, geometric alignment 512 optimizes depth- and / or parallax-based alignment to improve alignment. The quality of depth / parallax-based alignment and depth estimation is determined by a synthetic quality score. geometric The evaluation is performed, and the alignment, transformation, or method is adjusted to optimize the synthetic quality score. geometric The parameters.

[0071] Synthesis quality optimization 502 performs geometric alignment 512 by calculating the optimal 3D transformation to adjust the extrinsic parameters {A2, [R2|T2]} of c2 while keeping the extrinsic parameters {A1, [R1|T1]} of c1 unchanged, where A is the intrinsic parameter and [R|T] is the extrinsic rotation and translation. In one embodiment, synthesis quality optimization 502 during geometric alignment 512 calculates the optimal 3D transformation to perform geometric alignment 512 by directly searching the neighborhood of the extrinsic space [R2|T2] of c2. That is, synthesis quality optimization 502 searches for rotation and translation adjustments [R2, [R2|T2]]. optimal |T optimal ] * When applied to external parameters of c2, it leads to a maximized score. geometric For example, the optimal 3D transformation is calculated as follows:

[0072] [R optimal |T optimal ] * =argmaxscore geometric (search neighborhood of[R2,T2])

[0073] Where the search neighborhood of [R2, T2] is adjusted to cause c2 to rotate and translate around c1. The synthesis quality optimization during geometric alignment 512 502 results in different scores through computation. geometric Various transformations of the value and their application result in the maximum score. geometric The transformation of values ​​determines the optimal 3D transformation [R] optimal |T optimal ] * .

[0074] In another embodiment, during geometry alignment 512, the synthesis quality optimization 502 computes the optimal 3D transformation to perform geometry alignment 512 by reducing the search space through the optical flow vector. The relationship between the 3D synthesis space coordinates M and the projected 2D image space coordinates m is defined as follows:

[0075] m = sA([RT]M)

[0076] Where s is the depth scaling factor. The geometrically aligned 512 block estimates the optical flow vector between the 2D input 504 image from the source camera c1 and the 2D input 504 image from the target camera c2 using any estimation method (e.g., phase correlation, block-based methods, difference methods (e.g., Lucas-Kanade or Horn-Schunck), discrete optimization methods, or any other optical flow estimation technique). Once the geometrically aligned 512 block estimates the optical flow vector between the 2D input 504 image from the source camera c1 and the 2D input 504 image from the target camera c2, in the region where the 2D input 504 image from the source camera c1 and the 2D input 504 image from the target camera c2 overlap, the 3D transformation [R0|T0] of the target 2D image from c2 is estimated using the relationship between M and m, as described above. Due to optical flow vector estimation errors, the 3D transformation [R0|T0] may not achieve the optimal score. geometric As a result, the geometry-aligned 512 block applies the confidence value to the optical flow vector and evaluates the neighborhood of the 3D transform [R0|T0], where the neighborhood of the 3D transform [[R0|T0]] represents the reduced search neighborhood of [R2|T2]. The use of the geometry-aligned 512 block of the synthesis quality optimization 502 to estimate the optical flow vector results in a guided search direction and a reduced search space. In one embodiment, the optimal 3D transform is then computed as:

[0077] [R optimal |T optimal ] * =argmaxscore geometric (search neighborhood of[R0,T0]).

[0078] In one embodiment, time consistency is integrated into the score during optimization. geometric The 3D optimization transform in the model is a frame-time 3D volume optimization rather than a single-frame optimization.

[0079] If the quality score verification block 510 determines the photometric quality score calculated by the quality assessment block 508. photometric If the values ​​are below a threshold, then the synthesis quality optimization 502 performs photometric adjustment 514. In one embodiment, photometric adjustment 514 is a combination of data values ​​and software instructions that, when executed, adjust the color values ​​between the unstitched input 504 images to improve the photometric quality score. photometric .

[0080] In one embodiment, the photometric adjustment 514 attempts to match color intensity and color angle, as described above, between the 2D input 504 image from the source camera c1 and the 2D input 504 image from the target camera c2, in the overlapping region of the 2D input 504 image from the source camera c1 and the 2D input 504 image from the target camera c2. The photometric adjustment 514 calculates the optimal transform on each of the R, G, and B channels of the 2D image from the target camera C2. scale * ,optimal offset * ), so that:

[0081] target image = optimal scale *source image+optimal offset

[0082] In one embodiment, the photometric adjustment 514 directly searches for the offset and scale values, such that...

[0083] (optimal scale * ,optimal offset * =argmaxscore photometric (scale, offset)

[0084] Various values ​​are used for scale and offset. The photometric quality fraction (SCO) is maximized when the color values ​​of the 2D input 504 image from the source camera c1 are as close as possible to the color values ​​of the 2D input 504 image from the target camera c2.

[0085] In another embodiment, photometric adjustment 514 reduces the search space to improve performance by calculating an initial scale and initial offset using the above equations to begin the search. The initial scale calculated during photometric adjustment 514 is:

[0086]

[0087] The initial offset was calculated during photometric adjustment 514 as follows:

[0088] offset=mean(target image)-scale*mean(source image)

[0089] In one embodiment, if the quality score verification block 510 in the synthesis quality optimization 502 module determines all quality scores, as described above and in conjunction with the following... Figure 6Further described, above the threshold, the synthesis quality optimization 502 module outputs a 516 optimized 3D stitched image to the projection block, as described above. Figure 3 As described. In another embodiment, if the quality score verification 510 block determines one or more quality scores, such as score geometric or score photometric , or score geometric and score photometric If the value exceeds the threshold, the synthesis quality optimization module 502 outputs the optimized 3D stitched image 516 to the projection block.

[0090] Figure 6 This is a block diagram illustrating a quality assessment block 602 according to some embodiments of the present disclosure. The quality assessment block 602 calculates one or more quality scores 614, 618 associated with the input stitched image 606, as described above. Figure 5 In one embodiment, the quality assessment block 602 calculates one or more quality scores 614, 618 based at least in part on a comparison between the left image 604 and the stitched image 606, or the right image 608 and the stitched image 606. In another embodiment, the quality assessment block 602 calculates one or more quality scores 614, 618 based at least in part on a comparison between the left image 604 and the right image 608. In one embodiment, the quality assessment block 602 uses the left image 604, the right image 608, and the stitched image 606 to calculate one or more quality scores 614, 618. In one embodiment, the left image 604 represents the image from the left-facing camera c. L The 2D input image data is as further described herein. In one embodiment, the right image 608 is a representation from the right-facing camera c. R The data of the 2D input image.

[0091] The quality assessment block 602 uses the stitched image 606 to calculate the overlap 610 between two or more input images 604, 608. In one embodiment, the overlap 610 is data including information about regions in the coordinate space shared by the left image 604 and the right image 608 in the stitched image 606. In one embodiment, the coordinate space is a 3D projection space or a virtual camera space. In another embodiment, the coordinate space is a 2D projection space. In one embodiment, the coordinate space includes overlap 610 information from the left image 604 and the right image 608. In another embodiment, the coordinate space includes overlap 610 information from either the left image 604 or the right image 608.

[0092] In one embodiment, the quality assessment 602 block performs Qualcomm 612 operations to calculate the geometric score 614 or geometric quality score. geometric As mentioned above Figure 5 In one embodiment, the Qualcomm 612 operation is a data value and software instruction that, upon execution, uses one or more mathematical analysis techniques to calculate a geometric score 614, including objective quality measures and / or subjective quality measures. In one embodiment, the Qualcomm 612 operation determines the objective quality measure and calculates the geometric score 614 by averaging a high-frequency structural similarity index (SSIM) between the left image 604 or right image 608 and the stitched image 606. The SSIM is a perception-based model that treats image degradation as a perceptual change in structural information, while also incorporating important perceptual phenomena such as brightness masking and contrast masking terms. In another embodiment, the Qualcomm 612 operation determines the objective quality measure and calculates the geometric score 614 by generating a perceptual quality importance map (PQSM) between the left image 604 or right image 608 and the stitched image 606. The PQSM is an array whose elements represent the relative perceptual quality significance levels of corresponding regions and / or areas between the images. In one embodiment, the Qualcomm 612 operation uses root mean square error (RMSE), peak signal-to-noise ratio (PSNR), or any other objective quality metric that can be used to quantify the difference between the left image 604 or the right image 608 and the stitched image 606 to calculate the geometric score 614. In another embodiment, the Qualcomm 612 operation uses any subject quality metric (e.g., average opinion score) to calculate the geometric score 614.

[0093] In one embodiment, the quality assessment block 602 performs a low-pass operation 616 to calculate the photometric score 618 or the photometric quality score. photometric As mentioned above Figure 5 As described above. In one embodiment, the low-pass 616 operation is a data value and software instructions that, when executed, calculate a photometric score 618. In one embodiment, the low-pass 616 operation calculates the photometric score 618 as described above. Figure 5 As described herein. In another embodiment, the low-pass 616 operation is at least partially based on a spectral angle mapper (SAM) to calculate a photometric score 618 to determine photometric color quality and / or intensity amplitude ratio (IMR) to determine photometric intensity quality. In one embodiment, the low-pass 616 operation uses any other technique that can be used to quantify the color difference between the left image 604 or the right image 608 and the stitched image 606 to calculate the photometric score 618. In one embodiment, the low-pass 616 operation calculates the photometric score 618 based on a color space (such as RGB, CIELAB, CMYK, YIQ, YCbCr, YUV, HSV, HSL), or any other color space available for the left image 604 and the right image 608 captured by two or more cameras, as further described herein.

[0094] Now for reference Figure 7Each block of the method 700 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be executed by a processor that executes instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (standalone or in combination with another managed service) or plug-ins to another product, to name a few. Furthermore, by way of example, regarding... Figure 1 The video and / or image synthesis pipeline describes method 700, such as that performed by a surround-view system in an automotive application. However, these methods may be performed additionally or alternatively by any system or any combination of systems, including but not limited to those described herein.

[0095] Figure 7 This is a flowchart illustrating a method 700 for performing improved image compositing using compositing quality optimization according to some embodiments of the present disclosure. In block 702 of method 700, a compositing module or compositing pipeline receives two or more images from two or more cameras, along with calibration parameters associated with the two or more cameras used to generate the two or more images, as described above. Figure 1 As described.

[0096] At block 704, the compositing module or compositing pipeline transforms each input image into a compositing space, as described above. Figure 3 As described, an initial stitched image is generated in block 706. In one embodiment, blocks 704 and 706 are combined and the stitched image 706 is generated as a result of transforming two or more images into compositing space 704. In another embodiment, blocks 704 and 706 are executed independently and the stitched image 706 is generated by compositing module or compositing pipeline 704 after two or more 2D input images have been transformed into compositing space 704. In one embodiment, compositing module or compositing pipeline transforms two or more 2D input images into a single 3D virtual camera or stitching space using forward or reverse mapping. In another embodiment, compositing module or compositing pipeline transforms two or more 2D input images into a single 2D image.

[0097] In block 708, the quality assessment block of the synthesis module uses the above combination. Figure 6 The various techniques described calculate geometric fractions. In block 710, the quality assessment block of the synthesis module uses the above combination... Figure 5 and Figure 6The various techniques described calculate the photometric score. Based on the geometric score and / or photometric score calculated in blocks 708 and 710, the quality score verification block determines whether either the geometric score or the photometric score is below one or more thresholds. If neither the geometric score nor the photometric score is below one or more thresholds in 712, the compositing module or compositing pipeline converts the resulting stitched image to the projection space in block 718, as described above. Figure 3 As stated above.

[0098] If the geometric score is below the threshold 712, the synthesis module in block 714 reduces geometric distortion by calculating the optimal 3D geometric transformation, as described above. Figure 5 As described above, if the photometric score is below threshold 712, the synthesis module in block 716 reduces photometric distortion by applying one or more optimal photometric transformations, as described above. Figure 5 As described.

[0099] Example autonomous vehicles

[0100] Figure 8A This is an illustration of an example autonomous vehicle 800 according to some embodiments of the present disclosure. The autonomous vehicle 800 (or, alternatively, referred to herein as “vehicle 800”) may include, but is not limited to, passenger vehicles such as automobiles, trucks, buses, ambulances, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, engineering vehicles, underwater vessels, drones, and / or other types of vehicles (e.g., driverless and / or capable of accommodating one or more passengers). Autonomous vehicles are generally described according to the level of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of that standard). Vehicle 800 may be able to perform one or more functions that meet the requirements of Level 3 to Level 5 autonomous driving. For example, depending on the embodiment, vehicle 800 may be able to perform conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0101] Vehicle 800 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 800 may include a propulsion system 850, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 850 may be connected to the drivetrain of vehicle 800, which may include a transmission, to enable propulsion of vehicle 800. Propulsion system 850 may be controlled in response to receiving a signal from throttle / accelerator 852.

[0102] A steering system 854, which may include a steering wheel, can be used to steer the vehicle 800 (e.g., along a desired path or route) when the propulsion system 850 is operating (e.g., when the vehicle is in motion). The steering system 854 may receive signals from the steering actuator 856. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0103] The brake sensor system 846 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 848 and / or the brake sensor.

[0104] It may include one or more CPUs, one or more System-on-a-Chip (SoC) 804 ( Figure 8C One or more controllers 836, including one or more GPUs, may provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 800. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 848, to operate steering system 854 via one or more steering actuators 856, and / or to operate propulsion system 850 via one or more throttles / accelerators 852. One or more controllers 836 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 800. One or more controllers 836 may include a first controller 836 for autonomous driving functions, a second controller 836 for functional safety functions, a third controller 836 for artificial intelligence functions (e.g., computer vision), a fourth controller 836 for infotainment functions, a fifth controller 836 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 836 can handle two or more of the functions described above, and two or more controllers 836 can handle a single function, and / or any combination thereof.

[0105] One or more controllers 836 may provide signals for controlling one or more components and / or systems of vehicle 800 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, a global navigation satellite system sensor 858 (e.g., a Global Positioning System sensor), a RADAR sensor 860, an ultrasonic sensor 862, a LIDAR sensor 864, an inertial measurement unit (IMU) sensor 866 (e.g., an accelerometer, gyroscope, magnetic compass, magnetometer, etc.), a microphone 896, a stereo camera 868, a wide-angle camera 870 (e.g., a fisheye camera), an infrared camera 872, a surround camera 874 (e.g., a 360-degree camera), a long-range and / or medium-range camera 898, a speed sensor 844 (e.g., for measuring the rate of vehicle 800), a vibration sensor 842, a steering sensor 840, a braking sensor (e.g., as part of a braking sensor system 846), and / or other sensor types.

[0106] One or more controllers 836 may receive inputs (e.g., represented by input data) from the instrument cluster 832 of the vehicle 800 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 834, an auditory signaling device, a speaker, and / or via other components of the vehicle 800. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 8C Information such as the HD map 822, location data (e.g., the location of vehicle 800 on the map), direction, and the location of other vehicles (e.g., occupying a grid), as well as information about objects and their states perceived by the controller 836, etc. For example, the HMI display 834 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0107] The vehicle 800 further includes a network interface 824, which can communicate via one or more networks using one or more wireless antennas 826 and / or a modem. For example, the network interface 824 may be able to communicate via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 826 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0108] Figure 8BFor use in accordance with some embodiments of this disclosure Figure 8A This is an example of the camera position and field of view of an example autonomous vehicle 800. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 800.

[0109] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 800. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0110] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0111] One or more of the cameras can be mounted in mounting components such as custom-designed (3-D printed) parts to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0112] Cameras with a field of view that includes the environment in front of the vehicle 800 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 836 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining the preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (LDW), Autonomous Cruise Control (ACC), and / or other functions such as traffic sign recognition.

[0113] A variety of cameras can be used in front-facing configurations, including monocular camera platforms such as CMOS (Complementary Metal-Oxide-Semiconductor) color imagers. Another example could be a wide-angle camera 870, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 8B The image shows only one wide-angle camera, but any number of wide-angle cameras 870 can be present on the vehicle 800. Furthermore, a remote camera 898 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The remote camera 898 can also be used for object detection and classification, as well as basic object tracking.

[0114] One or more stereo cameras 868 may also be included in a front-mounted configuration. The stereo camera 868 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 868 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 868 may be used in addition to those described herein, or alternatively.

[0115] Cameras with a field of view including the side portion of the vehicle 800 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 874 (e.g., ... Figure 8BThe four surround cameras 874 shown can be mounted on the vehicle 800. The surround cameras 874 can include wide-angle cameras 870, fisheye cameras, 360-degree cameras, and / or the like. For example, four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 874 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0116] Cameras with a field of view that includes the environment behind the vehicle 800 (e.g., rear-view cameras) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range cameras 898, stereo cameras 868, infrared cameras 872, etc.).

[0117] Figure 8C For use in accordance with some embodiments of this disclosure Figure 8A The example autonomous vehicle 800 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0118] Figure 8C Each component, feature, and system in vehicle 800 is illustrated as being connected via bus 802. Bus 802 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 800 used to assist in the control of various features and functions of vehicle 800, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0119] Although bus 802 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 802 is represented by a single line, this is not intended to be limiting. For example, any number of buses 802 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 802 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 802 may be used for collision avoidance functions, and a second bus 802 may be used for drive control. In any example, each bus 802 may communicate with any component of vehicle 800, and two or more buses 802 may communicate with the same component. In some examples, each SoC 804, each controller 836, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 800) and may be connected to a common bus such as a CAN bus.

[0120] Vehicle 800 may include one or more controllers 836, such as those described herein. Figure 8A The controllers described herein. Controller 836 can be used for a wide variety of functions. Controller 836 can be coupled to any other different components and systems of vehicle 800 and can be used for the control of vehicle 800, artificial intelligence of vehicle 800, infotainment and / or the like for vehicle 800.

[0121] Vehicle 800 may include one or more System-on-Chip (SoC) 804s. SoC 804 may include a CPU 806, GPU 808, processor 810, cache 812, accelerator 814, data storage 816, and / or other components and features not shown. SoC 804 can be used to control vehicle 800 across a wide variety of platforms and systems. For example, one or more SoCs 804s may be combined with an HD map 822 in a system (e.g., the system of vehicle 800), the HD map being transmitted via a network interface 824 from one or more servers (e.g., [server name missing]). Figure 8D One or more servers (878) receive map refresh and / or updates.

[0122] The CPU 806 may include CPU clusters or CPU complexes (alternatively referred to herein as "CCPLEX"). The CPU 806 may include multiple cores and / or L2 cache. For example, in some embodiments, the CPU 806 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 806 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). The CPU 806 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of the CPU 806 can be active at any given time.

[0123] The CPU 806 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 806 can further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0124] The GPU 808 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). The GPU 808 may be programmable and efficient for parallel workloads. In some examples, the GPU 808 may use an enhanced tensor instruction set. The GPU 808 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, the GPU 808 may include at least eight streaming microprocessors. The GPU 808 may use a computer application programming interface (API). Furthermore, the GPU 808 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0125] In automotive and embedded applications, the GPU 808 can be power-optimized for optimal performance. For example, the GPU 808 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 808 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of L1 data cache and shared memory units to improve performance while simplifying programming.

[0126] The GPU 808 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0127] The GPU 808 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 808 to directly access the CPU 806 page tables. In such examples, when the GPU 808 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 806. In response, the CPU 806 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 808. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 806 and GPU 808, simplifying GPU 808 programming and porting applications to the GPU 808.

[0128] In addition, the GPU 808 may include access counters that track how frequently the GPU 808 accesses the memory of other processors. These access counters help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0129] The SoC 804 may include any number of caches 812, including those described herein. For example, cache 812 may include an L3 cache available to both the CPU 806 and the GPU 808 (e.g., it is connected to both the CPU 806 and the GPU 808). Cache 812 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0130] SoC 804 may include one or more arithmetic logic units (ALUs) that can be used to perform processing of any of a variety of tasks or operations related to vehicle 800, such as processing a DNN. Additionally, SoC 804 may include a floating-point unit (FPU) or other mathematical coprocessor or digital coprocessor type for performing mathematical operations within the system. For example, SoC 104 may include one or more FPUs integrated as execution units within CPU 806 and / or GPU 808.

[0131] SoC 804 may include one or more accelerators 814 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 804 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to complement GPU 808 and offload some tasks from GPU 808 (e.g., freeing up more cycles of GPU 808 to perform other tasks). As an example, accelerator 814 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0132] Accelerator 814 (e.g., hardware acceleration clusters) may include a Deep Learning Accelerator (DLA). A DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional trillion operations per second for deep learning applications and inference. TPUs may be accelerators configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. DLAs may be further optimized for a specific set of neural network types and floating-point operations as well as inference. DLAs are designed to provide higher performance per millimeter than general-purpose GPUs and significantly outperform CPUs. TPUs can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0133] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0134] The DLA can perform any function of the GPU 808, and by using inference accelerators, for example, a designer can make either the DLA or the GPU 808 target any function. For example, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 808 and / or other accelerators 814.

[0135] Accelerator 814 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0136] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0137] DMA enables PVA components to access system memory independently of the CPU 806. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0138] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0139] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Consequently, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0140] Accelerator 814 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 814. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects the PVA and DLA to memory.

[0141] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61808 standards, but other standards and protocols can also be used.

[0142] In some examples, SoC 804 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0143] Accelerators 814 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0144] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0145] In some examples, PVA can be used to perform intensive optical flow. For instance, PVA can be used to process raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide a processed RADAR signal before the next RADAR pulse is emitted. In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0146] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. This neural network can take at least a subset of parameters as input, such as bounding box dimensions, ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 866 outputs related to vehicle orientation and distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 864 or RADAR sensor 860), etc.

[0147] The SoC 804 may include one or more data storage units 816 (e.g., memory). The data storage unit 816 may be on-chip memory of the SoC 804, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, the data storage unit 816 may be large enough to store multiple instances of the neural network. The data storage unit 812 may include an L2 or L3 cache 812. References to the data storage unit 816 may include references to memory associated with the PVA, DLA, and / or other accelerators 814 as described herein.

[0148] SoC 804 may include one or more processors 810 (e.g., embedded processors). Processor 810 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 804 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 804 thermal and temperature sensor management, and / or SoC 804 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 804 may use the ring oscillator to detect the temperature of CPU 806, GPU 808, and / or accelerator 814. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 804 into a lower power state and / or place vehicle 800 into a driver-safe parking mode (e.g., safely stop vehicle 800).

[0149] The processor 810 may further include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0150] The processor 810 may further include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0151] The processor 810 may further include a security cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0152] The processor 810 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0153] The processor 810 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0154] Processor 810 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 870, the surround camera 874, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0155] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0156] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 808 does not need to continuously render new surfaces, the video image compositer can be further used in user interface components. Even when the GPU 808 is powered on and actively performing 3D rendering, the video image compositer can be used to offload the GPU 808 to improve performance and responsiveness.

[0157] The SoC 804 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions for receiving video and input from a camera. The SoC 804 may further include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0158] SoC 804 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 804 can be used to process data from cameras and sensors (e.g., LIDAR sensor 864, RADAR sensor 860, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 802 (e.g., vehicle 800 speed, steering wheel position, etc.), and data from GNSS sensor 888 (connected via Ethernet or CAN bus). SoC 804 may further include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine, and which can be used to free up CPU 806 from routine data management tasks.

[0159] The SoC 804 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 804 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 806, GPU 808, and data storage 816, the accelerator 814 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0160] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0161] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 820) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could further include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0162] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 808.

[0163] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 800. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 804 provides security against theft and / or carjacking.

[0164] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 896 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 804 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 858. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 862, a control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0165] The vehicle may include a CPU 818 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 804 via a high-speed interconnect (e.g., PCIe). The CPU 818 may include, for example, an x86 processor. The CPU 818 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 804, and / or monitoring the status and health of the controller 836 and / or the infotainment SoC 830.

[0166] Vehicle 800 may include a GPU 820 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 804 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 820 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs from sensors of vehicle 800 (e.g., sensor data).

[0167] Vehicle 800 may further include a network interface 824, which may include one or more wireless antennas 826 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 824 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 878 and / or other network devices), with other vehicles, and / or with computing devices (e.g., a passenger's client device). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 800 with information about vehicles approaching vehicle 800 (e.g., vehicles in front, to the side, and / or behind vehicle 800). This functionality can be part of vehicle 800's cooperative adaptive cruise control function.

[0168] Network interface 824 may include a SoC that provides modulation and demodulation functions and enables controller 836 to communicate over a wireless network. Network interface 824 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0169] Vehicle 800 may further include data storage 828, which may include off-chip (e.g., off-chip SoC 804) storage devices. Data storage 828 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0170] The vehicle 800 may further include GNSS sensors 858 (e.g., GPS and / or auxiliary GPS sensors) for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 858 may be used, including, for example, but not limited to, GPS sensors using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0171] Vehicle 800 may further include a RADAR sensor 860. The RADAR sensor 860 can be used by vehicle 800 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 860 can use CAN and / or bus 802 (e.g., to transmit data generated by the RADAR sensor 860) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 860 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0172] The RADAR sensor 860 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 860 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assist and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 800's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 800's lane.

[0173] As an example, a mid-range RADAR system can include a range of up to 860m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 850 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0174] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0175] Vehicle 800 may further include ultrasonic sensors 862. Ultrasonic sensors 862, which may be positioned at the front, rear, and / or sides of vehicle 800, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 862 can be used, and different ultrasonic sensors 862 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 862 can operate at functional safety level ASIL B.

[0176] Vehicle 800 may include a LIDAR sensor 864. The LIDAR sensor 864 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 864 may be of functional safety level ASIL B. In some examples, vehicle 800 may include multiple LIDAR sensors 864 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0177] In some examples, the LiDAR sensor 864 may be able to provide a list of objects and their distances within a 360-degree field of view. A commercially available LiDAR sensor 864 may have an advertising range of, for example, approximately 100m, with an accuracy of 2cm-3cm, and support for a 100Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 864 may be used. In such examples, the LiDAR sensor 864 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 800. In such examples, the LiDAR sensor 864 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. A front-mounted LiDAR sensor 864 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0178] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-browsing LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LiDAR, and because a flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 864 is less susceptible to motion blur, vibration, and / or shock.

[0179] The vehicle may further include an IMU sensor 866. In some examples, the IMU sensor 866 may be located at the center of the rear axle of the vehicle 800. The IMU sensor 866 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 866 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 866 may include an accelerometer, a gyroscope, and a magnetometer.

[0180] In some embodiments, the IMU sensor 866 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 866 can enable the vehicle 800 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 866 without input from a magnetic sensor. In some examples, the IMU sensor 866 and the GNSS sensor 858 can be combined into a single integrated unit.

[0181] The vehicle may include a microphone 896 placed in and / or around the vehicle 800. Among other things, the microphone 896 may be used for emergency vehicle detection and identification.

[0182] The vehicle may further include any number of camera types, including stereo camera 868, wide-angle camera 870, infrared camera 872, surround camera 874, long-range and / or mid-range camera 898, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 800. The camera types used depend on the embodiment and the requirements of the vehicle 800, and any combination of camera types can be used to provide the necessary coverage around the vehicle 800. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 8A and Figure 8B It was described in more detail.

[0183] Vehicle 800 may further include vibration sensor 842. Vibration sensor 842 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 842 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a freely rotating shaft).

[0184] Vehicle 800 may include ADAS system 838. In some examples, ADAS system 838 may include SoC. ADAS system 838 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0185] The ACC system can use a RADAR sensor 860, a LIDAR sensor 864, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 800 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 800 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0186] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or network connection (e.g., via the Internet) through network interface 824 and / or wireless antenna 826. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 800 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 800, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0187] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0188] An AEB (Autonomous Emergency Braking) system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0189] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0190] The LKA system is a variation of the LDW system. If vehicle 800 begins to leave the lane, the LKA system provides corrective steering input or braking to vehicle 800.

[0191] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can utilize a rear-facing camera and / or RADAR sensor 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0192] RCTW systems can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of a rear-view camera while the vehicle is reversing. Some RCTW systems include AEB (Autonomous Emergency Braking) to ensure the application of the vehicle's brakes to avoid a collision. RCTW systems may use one or more rear-view RADAR sensors 860 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0193] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as they alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in an autonomous vehicle 800, in the event of conflicting results, the vehicle 800 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 836 or the second controller 836). For example, in some embodiments, the ADAS system 838 may be a backup and / or auxiliary computer used to provide perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and varied software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system 838 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0194] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0195] The supervisory MCU can be configured to run a neural network trained and configured to determine the conditions under which the auxiliary computer provides a false alarm based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of and / or be included as components of the SoC 804.

[0196] In other examples, ADAS system 838 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware used by the host computer does not cause a substantial error.

[0197] In some examples, the output of ADAS system 838 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 838 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0198] Vehicle 800 may further include an infotainment SoC 830 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 830 may include a combination of hardware and software that can be used to provide vehicle 800 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 830 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 834, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 830 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from the ADAS system 838, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0199] The infotainment SoC 830 may include GPU functionality. The infotainment SoC 830 can communicate with other devices, systems, and / or components of the vehicle 800 via bus 802 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 830 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform autonomous driving functions in the event of a failure of the main controller 836 (e.g., the primary and / or backup computer of the vehicle 800). In such an example, the infotainment SoC 830 may place the vehicle 800 into a driver-safe parking mode as described herein.

[0200] Vehicle 800 may further include an instrument cluster 832 (e.g., a digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 832 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 832 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 830 and the instrument cluster 832. In other words, the instrument cluster 832 may be included as part of the infotainment SoC 830, or vice versa.

[0201] Figure 8D For cloud-based servers and according to some embodiments of this disclosure Figure 8A This is a system diagram illustrating communication between example autonomous vehicles 800. System 876 may include server 878, network 890, and vehicles including vehicle 800. Server 878 may include multiple GPUs 884(A)-884(H) (collectively referred to herein as GPU 884), PCIe switches 882(A)-882(H) (collectively referred to herein as PCIe switch 882), and / or CPUs 880(A)-880(B) (collectively referred to herein as CPU 880). GPUs 884, CPUs 880, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 886, such as, but not limited to, NVLink interface 888 developed by NVIDIA. In some examples, GPUs 884 are connected via NVLink and / or NVSwitch SoCs, and GPUs 884 and PCIe switches 882 are connected via PCIe interconnects. Although eight GPUs 884, two CPUs 880, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 878 may include any number of GPUs 884, CPUs 880, and / or PCIe switches. For example, each of the servers 878 may include eight, sixteen, thirty-two, and / or more GPUs 884.

[0202] Server 878 can receive image data from vehicles via network 890, representing images of unexpected or changed road conditions such as recently commenced roadworks. Server 878 can also transmit neural network 892, updated neural network 892, and / or map information 894, including information about traffic and road conditions, to vehicles via network 890. Updates to map information 894 may include updates to HD map 822, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 892, updated neural network 892, and / or map information 894 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 878 and / or other servers).

[0203] Server 878 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is unlabeled and / or unprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 890), and / or the machine learning model can be used by server 878 to remotely monitor the vehicle.

[0204] In some examples, server 878 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 878 may include a deep learning supercomputer powered by GPU 884 and / or a dedicated AI computer, such as the DGX and DGX station machines developed by NVIDIA. However, in some examples, server 878 may include a deep learning infrastructure in a data center that uses only CPU power.

[0205] The deep learning infrastructure of server 878 is capable of rapid, real-time inference and can be used to assess and verify the health of the processor, software, and / or associated hardware in vehicle 800. For example, the deep learning infrastructure can receive periodic updates from vehicle 800, such as image sequences and / or objects located within those image sequences by vehicle 800 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them to those identified by vehicle 800. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 800 has malfunctioned, then server 878 can transmit a signal to vehicle 800 instructing its fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0206] For inference, server 878 may include GPU 884 and one or more programmable inference accelerators (such as NVIDIA's TensorRT 3). The combination of GPU-powered servers and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0207] Example computing device

[0208] Figure 9 The following is a block diagram suitable for implementing some embodiments of the present disclosure: The computing device 900 may include an interconnect system 902 that is directly or indirectly coupled to the following devices: memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., a display), and one or more logic units 920.

[0209] although Figure 9 The various boxes are shown connected via an interconnect system 902 with wiring, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 918, such as a display device, can be considered an I / O component 914 (e.g., if the display is a touchscreen). As another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent a storage device other than the memory of GPU 908, CPU 906, and / or other components). In other words, Figure 9The computing devices mentioned are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," "augmented reality system," and / or other device or system types, as all of these are considered within the same category. Figure 9 Within the scope of computing devices.

[0210] Interconnect system 902 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 902 may include one or more link or bus types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Fast (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 906 may be directly connected to memory 904. Furthermore, CPU 906 may be directly connected to GPU 908. In cases where there is a direct or point-to-point connection between components, interconnect system 902 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required in computing device 900.

[0211] Memory 904 may include any of a wide variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by computing device 900. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media may include computer storage media and communication media.

[0212] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media, implemented in any way or by any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computing device 900. As used herein, computer storage media does not include the signal itself.

[0213] Computer storage media may contain computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transport medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0214] CPU 906 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. Each of CPU 906 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. CPU 906 may include any type of processor and may include different types of processors depending on the type of computing device 900 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor may be an advanced RISC mechanism (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, computing device 900 may also include one or more CPUs 906.

[0215] In addition to or replacing CPU 906, GPU 908 may also be configured to execute at least some computer-readable instructions to control one or more components of computing device 900 to perform one or more of the methods and / or processes described herein. One or more GPUs 908 may be integrated GPUs (e.g., having one or more CPUs 906) and / or one or more GPUs 908 may be discrete GPUs. In embodiments, one or more GPUs 908 may be coprocessors of one or more CPUs 906. Computing device 900 may use GPU 908 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPU 908 may be used for general-purpose computing on a GPU (GPGPU). GPU 908 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 908 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received via a host interface from CPU 906). GPU 908 may include graphics memory, such as display memory, for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 904. GPU 908 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or via a switch (e.g., using NVSwitch). When combined, each GPU 908 may generate different portions of pixel data or GPGPU data for different outputs (e.g., the first GPU for the first image, the second GPU for the second image). Each GPU may include its own memory or may share memory with other GPUs.

[0216] In addition to or replacing CPU 906 and / or GPU 908, logic unit 920 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 900 to perform one or more methods and / or processes described herein. In embodiments, CPU 906, GPU 908, and / or logic unit 920 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 920 may be part of and / or integrated into one or more CPUs 906 and / or one or more GPUs 908, and / or one or more logic units 920 may be discrete components of CPU 906 and / or GPU 908 or otherwise external thereto. In embodiments, one or more logic units 920 may be processors of one or more CPUs 906 and / or one or more GPUs 908.

[0217] Examples of logic unit 920 include one or more processing cores and / or components thereof, such as tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU)), application-specific integrated circuit (ASIC), floating-point unit (FPU), I / O element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0218] The communication interface 910 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 900 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 910 may include components and functions that enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0219] I / O port 912 enables computing device 900 to be logically coupled to other devices, including I / O component 914, presentation component 918, and / or other components, some of which may be built into (e.g., integrated into) computing device 900. Illustrative I / O component 914 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, browsers, printers, wireless devices, and so on. I / O component 914 can provide a Natural User Interface (NUI) for processing user-generated air gestures, voice, or other physiological input. In some instances, the input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 900 (described in more detail below). Computing device 900 may include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof for gesture detection and recognition. In addition, computing device 900 may include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 900 to render immersive augmented reality or virtual reality.

[0220] Power supply 916 may include hard-wired power supply, battery power supply, or a combination thereof. Power supply 916 may supply power to computing device 900 so that the components of computing device 900 can operate.

[0221] The presentation component 918 may include a display (such as a monitor, touch screen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 918 may receive data from other components (such as GPU 908, CPU 906, etc.) and output that data (e.g., as an image, video, sound, etc.).

[0222] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0223] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" could include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0224] The subject matter of this disclosure is described in detail herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “block” may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A method comprising: determining photometric and geometric differences in a first image and a second image within an overlap region between the first image and the second image; applying a first transformation to at least the overlap region between the first image and the second image to generate a temporal consistency score and integrate the temporal consistency score into at least one of the photometric and geometric differences, wherein the first transformation includes warping at least one of the first image and the second image according to other images within a time interval; and applying a second transformation to at least the first image, the second transformation including at least one of a photometric transformation based at least in part on the photometric difference and a geometric transformation based at least in part on the geometric difference.

2. The method of claim 1, further comprising: determining whether the second transformation includes the geometric transformation based at least in part on a first comparison of the geometric difference to a first threshold value; and determining whether the second transformation includes the photometric transformation based at least in part on a second comparison of the photometric difference to a second threshold value.

3. The method of claim 1, further comprising combining the first image and the second image into a third image based at least in part on the overlap region. the geometric difference includes at least one geometric quality score quantifying at least one error in alignment of the first image and the second image in the overlap region.

4. The method of claim 1, wherein, the photometric difference includes at least one photometric quality score quantifying a difference in color levels in the first image and the second image within the overlap region.

5. The method of claim 1, wherein, computing a geometric quality score quantifying at least one error in alignment of the first image and the second image in the overlap region, the geometric difference including the geometric quality score; 6. The method of claim 1, further comprising: comparing the geometric quality score to a threshold value; and based on the comparison, determining that the second transformation includes the geometric transformation.

7. The method of claim 1, further comprising: computing at least one photometric quality score quantifying a difference in color levels in the first image and the second image within the overlap region; comparing the photometric quality score to a threshold value; and based on the comparison, determining that the second transformation includes the photometric transformation.

8. A vehicle or robot implementing the method of claim 1, comprising: a first image capture device to capture the first image; a second image capture device to capture the second image; a display device to display a stitched image generated based at least in part on the transformed first image; one or more processors; and a memory including instructions executable by the one or more processors to cause the one or more processors to perform the method of claim 1.

9. A vehicle system, comprising: one or more processors; and a memory including instructions that, when executed by the one or more processors, cause the vehicle system to at least: determine an overlap region corresponding to an overlap between a first image and a second image; ​ ​ ​ ​ ​ computing a score indicative of at least one of photometric and geometric differences between the first and second images in the overlap region; applying a first transformation to at least the overlap region between the first and second images to generate a temporal consistency score and incorporate the temporal consistency score into the score, wherein the first transformation includes warping at least one of the first and second images according to other images within a time interval; and based at least in part on the score, applying a second transformation to at least the first image, the second transformation including at least one of a photometric transformation and a geometric transformation.

10. The vehicle system of claim 9, wherein, The memory further includes instructions that, in response to execution of the instructions by the one or more processors, cause the vehicle system to: determine whether the second transformation includes the geometric transformation based at least in part on a first comparison of the geometric differences to a first threshold; and determine whether the second transformation includes the photometric transformation based at least in part on a second comparison of the photometric differences to a second threshold.

11. The vehicle system of claim 10, wherein: the first image is associated with a first time and the second image is associated with a second time.

12. The vehicle system of claim 9, wherein the memory further includes instructions that, in response to execution of the instructions by the one or more processors, cause the vehicle system to: compute the score based at least in part on optical flow vectors available to facilitate identification of one or more object differences between the first and second images in the overlap region; and apply one or more geometric transformations to at least the first image as a result of the one or more object differences.

13. The vehicle system of claim 9, wherein, The memory further includes instructions that, in response to execution of the instructions by the one or more processors, cause the vehicle system to: compute the score based at least in part on one or more color differences between each color channel of the first and second images in the overlap region; and apply one or more photometric transformations to the first or second image as a result of the one or more color differences.

14. The vehicle system of claim 9, wherein, The memory further includes instructions that, in response to execution of the instructions by the one or more processors, cause the vehicle system to: infer a second score from the score using one or more neural networks; and transform at least the first image based at least in part on the second score.

15. The vehicle system of claim 9, wherein, The memory further includes instructions that, in response to execution of the instructions by the one or more processors, cause the vehicle system to combine the first and second images into a third image based at least in part on the overlap region.

16. The vehicle system of claim 9, wherein, The memory further includes instructions that, in response to execution of the instructions by the one or more processors, cause the vehicle system to: compute a second score as a result of applying the second transformation to at least the first image; compute one or more transformations to apply to at least the first image based at least in part on the second score; and apply the one or more transformations to at least the first image.

17. The vehicle system of claim 9, wherein, The memory further includes instructions that, in response to execution of the instructions by the one or more processors, cause the vehicle system to: combine, as a result of the score being above a threshold, the first image and the second image into a third image based at least in part on the overlap region.

18. The vehicle system of claim 9, further comprising: one or more controllers; a network interface; a display; a propulsion system; and two or more surround cameras operable to capture the first image and the second image.

19. A robotic system, comprising: one or more processors coupled to a computer-readable medium; the computer-readable medium storing executable instructions that, in response to execution of the instructions by the one or more processors, cause the system to: determine an overlap region corresponding to an overlap between a first image and a second image; compute a score indicative of at least one of photometric and geometric differences between the first image and the second image in the overlap region; apply a first transformation to at least the overlap region between the first image and the second image to generate a temporal consistency score and integrate the temporal consistency score into the score, wherein the first transformation includes warping at least one of the first image and the second image according to other images within a time interval; and apply, based at least in part on the score, a second transformation to at least the first image, the second transformation including at least one of a photometric transformation and a geometric transformation.

20. The robotic system of claim 19, wherein the computer-readable medium further includes instructions that, if executed by the one or more processors, cause the one or more processors to: compute a second score as a result of applying the second transformation to at least the first image; and compute one or more transformations to apply to at least the first image based at least in part on the second score; and apply the one or more transformations to at least the first image.

21. The robotic system of claim 19, wherein the computer-readable medium further includes instructions that, if executed by the one or more processors, cause the one or more processors to: infer a second score from the score using one or more neural networks; and transform at least the first image based at least in part on the second score.

22. The robotic system of claim 19, wherein the computer-readable medium further includes instructions that, if executed by the one or more processors, cause the one or more processors to: apply one or more three-dimensional transformations to at least the first image as a result of the second transformation including the geometric transformation.

23. The robotic system of claim 19, wherein the computer-readable medium further includes instructions that, if executed by the one or more processors, cause the one or more processors to: compute the score based at least in part on one or more color differences between the first image and the second image; and ​ adjusting one or more color parameters of at least the first image as a result of the one or more color differences.

24. The robotic system of claim 19, wherein the computer-readable medium further comprises instructions that, if executed by the one or more processors, cause the one or more processors to: transform at least the first image into a third image; determine a second overlap region corresponding to a second overlap between a fourth image and a fifth image; compute a second score indicative of at least one of a photometric difference and a geometric difference between the fourth image and the fifth image in the second overlap region; transform at least the fourth image into a sixth image based at least in part on the second score; and combine the third image and the sixth image to create video data.

25. The robotic system of claim 19, wherein the computer-readable medium further comprises instructions that, if executed by the one or more processors, cause the one or more processors to: determine whether the second transform includes the geometric transform based at least in part on a first comparison of the geometric difference to a first threshold; and determine whether the second transform includes the photometric transform based at least in part on a second comparison of the photometric difference to a second threshold.

26. The robotic system of claim 19, wherein: the first image is associated with a first time and the second image is associated with a second time within a time interval; and the score is indicative of a temporal difference between the first image and the second image within the time interval.

27. The robotic system of claim 19, further comprising: one or more robotic arms; one or more servo controllers; one or more sensors; a communication interface; and two or more cameras operable to capture at least the first image and the second image. ​

Images