Any world view generation

Abstract

A method and system for rendering an output image from multiple input images. A plurality of input images are received, each of the input images being captured from a different first position. A view specification for rendering the output image is received, the view specification including at least a second position. The second position is different from each of the first positions. An output image is rendered based at least in part on the plurality of input images and the view specification, the output image including an image of the area as viewed from the second position. The output image is displayed on a display.

Description

Technical Field

[0001] The field of the invention generally relates to perspective view generation and image rendering. Background Art

[0002] Many types of vehicles utilize one or more cameras to capture, display, and / or record images and / or video of the vehicle's surroundings to assist with driving, navigation, or other tasks. Furthermore, cameras are widely used in a variety of other practical applications for surveillance, security, or entertainment to record images and / or video of a specific environment, among other possibilities. However, for practical reasons, it can be difficult or costly to mount a camera in a specific location from which images or video are desired. Therefore, improvements in this area are desirable. Summary of the Invention

[0003] Various embodiments of any world view (AWW) systems and methods for rendering an output image from multiple input images are described, where the output image is from an angle corresponding to a position that is different from each of the multiple positions from which the input images were taken.

[0004] In some embodiments, a plurality of input images are received, and each input image is taken from a different first position. A view specification for rendering an output image may be received, and the view specification includes at least a second position. In some embodiments, the first position and the second position may each be specified by three spatial coordinates (e.g., (x, y, z)) in a global coordinate system. The view specification may also include one or more of a pose specification, the pose specification including an azimuth and / or polar angle, a field of view (e.g., zoom or magnification), a projection surface depth, one or more projection surface parameters, a pixel resolution, and a blend width.

[0005] The second location can be different from each of the first locations. An output image can be rendered based at least in part on the plurality of input images and the view specification, and the output image can include an image of the area as seen from the second location. For example, the second location can be a "virtual location" for rendering the output image, even though there may not be a camera at that location.

[0006] In some embodiments, determining a projection surface and rendering an output image based at least in part on the plurality of input images includes mapping pixels of the plurality of input images onto the projection surface and mapping locations on the projection surface to pixels of the output image. The projection surface may be determined to approximate the surroundings of a camera used to acquire the plurality of input images.

[0007] In some embodiments, rendering an output image based at least in part on a plurality of input images and a view specification includes, for each of a plurality of pixel blocks of the output image, determining a mapping between a plurality of pixels along a perimeter of the corresponding pixel block and a plurality of corresponding pixels of each of one or more of the input images. For example, the plurality of pixels along the perimeter of the corresponding pixel block may be four corner pixels of a square pixel block. For each of the one or more input images and for each of the plurality of pixel blocks of the output image, a projection mapping may be performed on pixels within the perimeter of the corresponding pixel of the input image to the corresponding pixel in the output image, and the projection mapping may be based at least in part on the mapping between the plurality of pixels along the perimeter of the corresponding pixel block and the plurality of corresponding pixels of the one or more input images.

[0008] The output image can be displayed on a display. In some embodiments, a series of output images are repeatedly rendered based on a series of corresponding multiple input images, and the series of output images are displayed in real time on the display to produce real-time video. The output image and / or output video can be used as input to another application (e.g., for further processing), and / or the output can be stored on a storage medium.

[0009] This summary is intended to provide a brief overview of some of the topics described in this document. It will be appreciated, therefore, that the features described above are merely examples and should not be construed as narrowing the scope or spirit of the subject matter described herein in any way. Further features, aspects, and advantages of the subject matter described herein will become apparent from the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present invention may be better understood when the following detailed description of the preferred embodiments is considered in conjunction with the following drawings, in which:

[0011] Figure 1 is a schematic diagram of an advanced driver assistance system (ADAS) platform according to some embodiments;

[0012] Figure 2A is a graphical representation of raw camera data acquired from four cameras attached to a vehicle, according to some embodiments;

[0013] Figure 2B According to some embodiments, Figure 2A Illustration of dual-view output images produced by the raw camera data shown;

[0014] Figure 2C is a view from above and behind the vehicle according to some embodiments. Figure 2A Illustration of the output image produced by the raw camera data shown;

[0015] Figure 3 is a flow chart illustrating a method for rendering an output image from multiple input images according to some embodiments;

[0016] Figure 4 is a flow chart illustrating a method for determining a projection surface according to some embodiments;

[0017] Figure 5 is a flow chart illustrating a method for performing tile-by-tile output image rendering according to some embodiments;

[0018] Figure 6 is a schematic diagram illustrating a high-level data flow for calibrating and rendering output images from multiple input images according to some embodiments;

[0019] Figure 7 is an illustration of conformal and pinhole camera projection models according to some embodiments;

[0020] Figure 8A and Figure 8B is an illustration of an original conformal (fisheye) lens image and a reprojected image using a pinhole camera model according to some embodiments;

[0021] Figure 9 is an illustration of the reprojection of an object point in a world coordinate system to an image point according to some embodiments;

[0022] Figure 10 is an illustration of a reprojection of a view image point to a source image according to some embodiments;

[0023] Figure 11 is a graphical representation of the depth of an object point on a ground plane according to some embodiments;

[0024] Figure 12 is an illustration of a cylindrical projected surface model of a vehicle's surroundings according to some embodiments;

[0025] Figure 13A and Figure 13B illustrates a grid of view image tiles mapped to two fisheye source images according to some embodiments;

[0026] Figure 14 According to some embodiments, JavaScript TM Pseudocode of the code structure for BlockInfo in Object Notification (JSON) file format;

[0027] Figure 15 is a pseudocode illustrating the code structure for BlockInfo as a C / C++ structure according to some embodiments;

[0028] Figure 16 is a pseudo code illustrating the code structure for CamData in JSON file format according to some embodiments;

[0029] Figure 17 is a top view of the nominal orientation of a world coordinate system according to some embodiments;

[0030] Figure 18 is a pseudo code illustrating the code structure for ViewSpec in JSON file format according to some embodiments;

[0031] Figure 19 is a graphical representation of azimuth and elevation angles relative to a camera on a car, according to some embodiments;

[0032] Figure 20 illustrates a first verification image from a camera calibration process according to some embodiments;

[0033] Figure 21 illustrates a second verification image from a camera calibration process according to some embodiments;

[0034] Figure 22 is a rear-view output image rendered from three source cameras according to some embodiments;

[0035] Figure 23 is a parallel processing hardware diagram illustrating resource allocation for a two-chip design according to some embodiments;

[0036] Figure 24 is a parallel processing hardware diagram illustrating resource allocation for chip three according to some embodiments;

[0037] Figure 25 is a schematic data flow diagram for a three-chip design according to some embodiments;

[0038] Figure 26 is a schematic data flow diagram for Chip 1 and Chip 2 according to some embodiments;

[0039] Figure 27 is a schematic diagram of the internal structure of a demosaicing group cell according to some embodiments;

[0040] Figure 28 is a schematic data flow diagram for chip 3 according to some embodiments;

[0041] Figure 29 is a schematic data flow diagram for photometric alignment according to some embodiments;

[0042] Figure 30 is a block diagram illustrating one embodiment of a multi-processor system (MPS) according to some embodiments;

[0043] Figure 31 is a block diagram illustrating one embodiment of an MPS connection scheme according to some embodiments;

[0044] Figure 32 is a more detailed diagram of one embodiment of an MPS architecture according to some embodiments; and

[0045] Figure 33 The diagram illustrates the Figure 32 An example MPS consisting of PEs (squares) evenly interspersed with DMRs (circles) is shown in FIG.

[0046] While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and are herein described in detail. However, it should be understood that the drawings and detailed description thereof are not intended to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims. DETAILED DESCRIPTION

[0047] Incorporated by Reference

[0048] The following references are incorporated herein by reference in their entirety as if fully and completely set forth herein:

[0049] the term

[0050] The following is a glossary of terms used in this application:

[0051] Memory medium - any of various types of memory devices or storage devices. The term "memory medium" is intended to include installation media, such as CD-ROMs, floppy disks, or tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; or non-volatile memory, such as magnetic media, such as hard drives, optical storage devices, or ROM, EPROM, FLASH, etc. Memory media may also include other types of memory, or a combination thereof. Additionally, a memory medium may be located in a first computer in which a program is executed, and / or may be located in a second, different computer connected to the first computer via a network, such as the Internet. In the latter case, the second computer may provide program instructions to the first computer for execution. The term "memory medium" may include two or more memory media, which may reside in different locations, for example, in different computers connected via a network.

[0052] Carrier Medium—storage media as described above, as well as physical transmission media such as a bus, network, and / or other physical transmission media that convey signals such as electrical or optical signals.

[0053] Programmable hardware elements - include various hardware devices that include multiple programmable function blocks connected via programmable or hardwired interconnects. Examples include FPGAs (field programmable gate arrays), PLDs (programmable logic devices), FPOAs (field programmable object arrays), and CPLDs (complex PLDs). Programmable function blocks can range from fine-grained (combinatorial logic or lookup tables) to coarse-grained (arithmetic logic units or processor cores). Programmable hardware elements may also be referred to as "reconfigurable logic."

[0054] Application-Specific Integrated Circuit (ASIC)—This term is intended to have the full breadth of its ordinary meaning. The term ASIC is intended to include integrated circuits customized for a specific application, rather than general-purpose programmable devices, although ASICs may include programmable processor cores as building blocks. Cell phone processors, MP3 player chips, and many other single-function ICs are examples of ASICs. ASICs are typically described in a hardware description language such as Verilog or VHDL.

[0055] Program - The term "program" is intended to have the full breadth of its ordinary meaning. The term "program" includes 1) a software program that can be stored in a memory and executed by a processor, or 2) a hardware configuration program that can be used to configure a programmable hardware element or ASIC.

[0056] Software Program - The term "software program" is intended to have the full breadth of its ordinary meaning and includes any type of program instructions, code, scripts, and / or data, or combinations thereof, that can be stored in a memory medium and executed by a processor. Exemplary software programs include programs written in text-based programming languages, e.g., imperative or procedural languages ​​such as C, C++, PASCAL, FORTRAN, COBOL, JAVA, assembly language, etc.; graphical programs (programs written in a graphical programming language); assembly language programs; programs that have been compiled into machine language; scripts; and other types of executable software. A software program may include two or more software programs that interoperate in some manner.

[0057] Hardware Configuration Program - a program, such as a netlist or bit file, that can be used to program or configure a programmable hardware element or ASIC.

[0058] Computer system - any of various types of computing or processing systems, including personal computer systems (PCs), mainframe computer systems, workstations, network appliances, Internet appliances, personal digital assistants (PDAs), grid computing systems, or other devices or combinations of devices. In general, the term "computer system" can be broadly defined as any device (or combination of devices) that includes at least one processor that executes instructions from a memory medium.

[0059] Automatically refers to actions or operations performed by a computer system (e.g., software executed by a computer system) or a device (e.g., a circuit system, a programmable hardware element, an ASIC, etc.) without a user directly specifying or executing the action or operation. Thus, the term "automatically" is in contrast to operations that are manually performed or specified by a user, where the user provides input to directly perform the operation. An automatic process can be initiated by user-provided input, but the subsequent actions performed "automatically" are not user-specified, i.e., they are not "manually" performed where the user specifies each action to be performed. For example, a user filling out an electronic form by selecting each field and providing input specifying information (e.g., by typing information, selecting checkboxes, radio buttons, etc.) is manually filling out the form, even though the computer system can update the form in response to user actions. A form can be automatically filled out by a computer system, where the computer system (e.g., software executing on the computer system) analyzes the fields of the form and fills it out without any user input specifying the answers to the fields. As described above, a user can invoke automatic filling of a form without participating in the actual filling out of the form (e.g., the user does not manually specify the answers to the fields, but rather, the fields are automatically completed). This specification provides various examples of operations that are automatically performed in response to actions that a user has taken.

[0060] Definition of abbreviations

[0061]

[0062]

[0063] Detailed description

[0064] introduce

[0065] Embodiments herein provide systems and methods for generating output images and / or videos rendered in any world view, which utilize multi-view geometry transforms of frames of images and / or videos from multiple cameras to synthesize a virtually rendered video of an environment, as if captured by a floating virtual camera. In some embodiments, the virtual camera can have six degrees of freedom, including: x, y, z position; azimuth and elevation poses; and field of view angle. Advantageously, the virtual camera can be located at a position different from each of the multiple cameras. Additionally, the projection surface used to render the output images / video can be dynamic and linked to distance estimates of objects surrounding the virtual camera. Distance estimates can be obtained using ranging sensors such as ultrasonic or mmWave ranging sensors, radar, lidar, or using computer vision techniques applied to video frames. Embodiments herein can be used in a variety of applications, such as in automotive advanced driver assistance systems (ADAS), drone video capture, office or home environments, or other platforms with multiple cameras.

[0066] In some embodiments, multiple camera views are combined into a single perspective that provides the operator with control over a virtual viewpoint synthesized from multiple cameras positioned around the perimeter of a vehicle or other object. This composite view can integrate multiple separate camera streams while seamlessly blending distortion- and lighting-corrected images from each camera. Additionally, the user can specify a perspective or virtual viewpoint to display, for example, a bird's-eye view from above the vehicle.

[0067] In some embodiments, a HyperX TM Parallel processing architectures such as NVIDIA NVIDIA DRIVE technology can be used to accommodate the high computational cost of executing algorithms that ingest multimodal sensor data and provide information to the driver and vehicle systems. Advantageously, a virtual any-world view can be rendered, providing a 360° surround view with dynamic virtual camera positions and dynamic projection surfaces. According to some embodiments, Figure 1 A conceptual diagram of the functional relationships between the components of any world view system is shown in . Advantageously, some embodiments herein create a system that helps a driver operate a vehicle in a safe manner when the view around the vehicle is limited by the structure of the vehicle itself or by the ability to physically visualize all directions around the driver. Methods and systems are described herein for converting video data captured from multiple digital video cameras into rendered video displays representing virtual viewpoints around a vehicle or other object. In some embodiments, the rendered output image(s) can be used as input to another application (e.g., for further processing) and / or can be stored in memory or on another type of storage medium.

[0068] Figure 3 -Rendering output image flow chart

[0069] Figure 3 is a flowchart illustrating a method for rendering an output image from multiple input images according to some embodiments. In some embodiments, the described method can be performed by a computer system including a processor coupled to a multi-camera system. The processor can be coupled to one or more non-transitory computer-readable storage media, which can have program instructions stored thereon for performing the described method. In some embodiments, and as described in more detail in Section 5, the processor can be a parallel processing hardware system that can be used to reduce the latency of computational steps. In various embodiments, some elements of the illustrated method can be performed simultaneously in an order different from that shown, can be replaced by other method elements, or can be omitted. Additional method elements can also be performed as needed. As shown, the method can operate as follows.

[0070] At 302, a plurality of input images is received, wherein each of the plurality of input images is captured from a different first position. In other words, each input image may be captured by a camera or another type of image acquisition device located at a different first position, such that the plurality of input images correspond to a plurality of respective first positions. The cameras may be mounted at various locations on a vehicle, such as a car, truck, airplane, or boat; a drone; furniture; or any other type of object. Each of the plurality of cameras may have its own position, orientation, and / or field of view, and the fields of view of the respective cameras in the plurality of cameras may partially overlap.

[0071] At 304, a view specification is received to render an output image. The view specification may be received in response to user input. For example, a user may present input to determine one or more parameters of a view specification that may be provided to the computing system for rendering the output image. The view specification may include a second position for rendering the output image, and the second position may be different from each first position. As an example, if the plurality of input images includes four input images taken from four different first positions on a vehicle (e.g., Figure 2A), then the second position can be located above the vehicle or at another location outside the vehicle. The view specification can also specify one or more additional parameters for rendering the output image, including but not limited to azimuth, polar angle, field of view, projection surface depth, one or more projection surface parameters, pixel resolution and / or blending width. Note that the azimuth and polar angles are intended to be measured in a spherical coordinate system with the origin as the second position. The azimuth and polar angles can thus specify the view orientation for rendering the output image from the perspective of the second position. The field of view can specify the solid angle and / or magnification of the output image. In addition, the view specification can specify the pixel resolution of the output image (e.g., 1280x720, 1920x1080 or other resolution) and / or the blending width for blending the contributions of overlapping portions of different input images to the output image. The projection surface depth can specify one or more depth parameters of the projection surface for rendering the output image. For example, for an embodiment in which the projection surface is an elliptical cylinder, the view specification can specify one or both of the major axis and the minor axis of the elliptical cylinder. One or more projection surface parameters can also specify different sizes, shapes or presets of the projection surface. Alternatively, and as described in more detail below, the projection surface may be automatically determined by a computing system.

[0072] For embodiments in which an output video is generated by rendering a series of output images based on a series of corresponding multiple input images, a user can dynamically change the view specification to change one or more parameters of the output video in real time. For example, a user can adjust one or more parameters of the view specification, such as the second position and / or viewpoint direction, to smoothly change the viewpoint of the output video in real time. As an example, this can be advantageous in a parking scenario, where a user can change the viewing angle of the output video by changing one or more parameters of the view specification during parking.

[0073] In some embodiments, further user input specifying one or more second view specifications is received, and one or more second output images are rendered based at least in part on the one or more second view specifications and the plurality of input images. In other words, a single set of input image data can be used to render multiple different output images from different viewpoints described by the first view specification and the second view specification. The one or more second output images can be displayed on the same display as the first output image (i.e., the output image generated from the first view specification), or they can be displayed on one or more different displays.

[0074] At 306, an output image is rendered based at least in part on the plurality of input images and the view specification. The output image is an image of the region as viewed from the second position and according to one or more other parameters of the view specification. In some embodiments, as described below with reference to Figure 4As described in more detail, the output image is further rendered based on the projection surface. Additionally or alternatively, the output image may be rendered block by block, as described below with reference to Figure 5 Advantageously, the output image may be rendered from a viewpoint (ie, position, azimuth, and / or polar angle, among other potential characteristics of the view specification) that is different from that of any of the plurality of input images.

[0075] In some embodiments, at least a portion of the output image is derived from two or more input images from a plurality of input images, and rendering at least a portion of the output image includes blending contributions from the two or more input images from the plurality of input images. For example, two of the input images may partially overlap in their fields of view, such that pixels from the two input images within the overlapping region may contribute to corresponding portions of the output image.

[0076] In some embodiments, blending is weighted based at least in part on the proximity of a first position of two or more input images to a projected position of at least a portion of an output image. For example, when two of the input images overlap within a portion of their respective fields of view, a determination can be made as to how far the projected position of the overlapping region is from the first position from which each of the two input images was captured. In this example, the contributions of the two input images to corresponding portions of the output image can be weighted such that an input image captured from a first position closer than the projected position is given a greater weight than another input image captured from a first position farther away. Additionally or alternatively, blending can correct for differences in illumination between the two or more input images.

[0077] At 308, the output image is displayed on a display. The display may be collocated with the computing device that renders the output image, or they may be remotely located and connected via a wireless network. In some embodiments, the display may be installed within a vehicle to provide ADAS services to the driver or operator of the vehicle. Alternatively, the display may be located remotely from the vehicle and may be included in a device configured to remotely drive or operate the vehicle.

[0078] In some embodiments, a series of output images may be repeatedly rendered based on a series of corresponding multiple input images, and the series of output images may be displayed on a display in real time to produce real-time video. In other words, although reference is made to Figure 3 The described methods illustrate a single output image rendered from multiple input images, but it is within the scope of some embodiments to generate an output video from multiple input videos, where the method steps described herein are repeated for each frame of the video.

[0079] Figure 4 -Projection Surface Flowchart

[0080] Figure 4 is a flowchart illustrating a method for determining a projection surface to assist in rendering an output image according to some embodiments. Figure 3 and / or Figure 5 Describe the method steps to perform Figure 4 The method steps described in , as part of a method for rendering an output image from multiple input images. In some embodiments, the described method can be performed by a computer system including a processor coupled to a multi-camera system. The processor can be coupled to one or more non-transitory computer-readable storage media, which can have program instructions stored thereon for performing the described method. In some embodiments, and as described in more detail in Section 5, the processor can be a parallel processing hardware system, which can be used to reduce the latency of the computational steps. In various embodiments, some elements of the illustrated method can be performed simultaneously in an order different from the order shown, can be replaced by other method elements, or can be omitted. Additional method elements can also be performed as needed. As shown, the method can operate as follows.

[0081] At 402, a geometric layout of an environment is determined. For example, a geometric layout of an environment surrounding a first location associated with a plurality of input images can be determined. The geometric layout can be understood as an approximation of the surrounding environment representing the first location, which can be used to determine an effective projection surface for rendering an output image. Note that step 302 is optional, and the projection surface can be determined based on one or more other factors without determining the geometric layout of the environment. For example, as described below, the projection surface can be an elliptical cylinder by default, and the dimensions of the elliptical cylinder (e.g., the length of the major axis and / or minor axis) can be determined based on the speed of the vehicle or other factors (such as, for example, the distance to nearby objects).

[0082] In some embodiments, determining the geometric layout of the environment includes determining ranging information associated with an area surrounding the first location, and determining the projection surface is performed at least in part based on the ranging information. For example, one or more ranging devices can be juxtaposed with one or more of the multiple cameras used to acquire the multiple input images. The (one or more) ranging devices can utilize radar, sonar, lidar, computer vision processing algorithms, machine learning, and / or other techniques to estimate the distance from the corresponding first location to various portions of the corresponding input image. For example, if a particular input image shows a road, another vehicle, and / or a tree or other object, the ranging device can estimate the distance from the first location of the particular input image to the road, vehicle, tree, or other object.

[0083] At 404, a projection surface is determined. The projection surface may include a basic spline surface that approximates the environment surrounding the first location associated with the plurality of input images. In some embodiments, estimated distances generated by one or more ranging devices may be used to determine the projection surface. For example, a projection surface derived from distances obtained by the ranging device(s) may be determined. In other words, a projection surface may be determined that approximates the environment surrounding a vehicle or other object on which the plurality of cameras are mounted, and ranging information may be used to inform the approximation.

[0084] In some embodiments, the projected surface can be a segmented and / or discontinuous surface determined based on the vehicle's environment or other factors. In other words, the projected surface can be determined to any desired level of specificity or detail, up to and including a pixel-by-pixel depth specification of the projected surface's position relative to the origin for any desired solid angle around the origin (e.g., up to and including 4π steradians).

[0085] In some embodiments, the first location is a location on a vehicle, and the projection surface includes a horizontal portion surrounding the vehicle and a vertical portion surrounding the horizontal portion. In some embodiments, the horizontal portion of the projection surface is an elliptical disk surrounding the vehicle, the vertical portion of the projection surface is an elliptical cylinder surrounding the elliptical disk, and the projection surface also includes an intermediate region smoothly joining the elliptical disk and the elliptical cylinder. In an ADAS embodiment, where the camera used to capture the input image is mounted on the first vehicle, the major axis of the elliptical disk can extend from the front of the vehicle to the rear of the vehicle, and the minor axis of the elliptical disk can extend from the left side of the vehicle to the right side of the vehicle. In some embodiments, the elliptical cylindrical shape of the projection surface can be modified to accommodate one or more objects determined to be proximate to the first vehicle (e.g., "proximate" in this context can be understood to mean within the vertical portion of the elliptical cylinder). In other words, if an object, such as a second vehicle, is determined to be proximate to the first vehicle through a range finding procedure or other mechanism, the elliptical cylindrical projection surface can be modified to include an approximation of that location and the shape of the second vehicle.

[0086] In some embodiments, the size of the horizontal portion is determined at least in part based on the vehicle's current speed, where a higher current speed of the vehicle determines a larger horizontal portion than a lower current speed of the vehicle. In some embodiments, the size of the horizontal portion is determined at least in part based on the shortest distance to nearby objects. For example, the size of the horizontal portion can be scaled based on the proximity of another vehicle or other object to the plurality of cameras. In other words, and as an example, as a vehicle approaches a vehicle on which the plurality of cameras generating the input images are mounted, the size of the horizontal portion can be reduced to match the distance to the approaching vehicle.

[0087] In some embodiments, one or both of steps 402 and 404 may be performed as part of a calibration process. In other words, steps 402 and 404 may be performed during the design or production of a computing system designed to render output images.

[0088] At 406, an output image is rendered based at least in part on the projection surface. In some embodiments, rendering the output image includes mapping pixels of the plurality of input images onto the projection surface, and mapping locations on the projection surface to pixels of the output image. For example, each of the plurality of input images may be projected onto the projection surface to produce a rendered environment, and the output image may be rendered by determining how the rendered environment on the projection surface would appear if viewed from the perspective described by the view specification.

[0089] Figure 5 - Chunk-specific rendering flow graph

[0090] Figure 5 is a flowchart illustrating a method for dividing an output image into a plurality of pixel blocks to assist in rendering the output image according to some embodiments. Figure 3 and / or Figure 4 Describe the method steps to perform Figure 5 The method steps described in , as part of a method for rendering an output image from multiple input images. In some embodiments, the described method can be performed by a computer system including a processor coupled to a multi-camera system. The processor can be coupled to one or more non-transitory computer-readable storage media, which can have program instructions stored thereon for performing the described method. In some embodiments, and as described in more detail in Section 5, the processor can be a parallel processing hardware system that can be used to reduce the latency of the computational steps. In various embodiments, some elements of the illustrated method can be performed simultaneously in an order different from the order shown, can be replaced by other method elements, or can be omitted. Additional method elements can also be performed as needed. As shown, the method can operate as follows.

[0091] At 502, in some embodiments, rendering an output image includes performing the rendering separately for each of a plurality of pixel blocks of the output image. For example, the output image can be divided into a plurality of pixel blocks of any desired size and / or shape, and rendering the output image can be performed block by block. In some embodiments, the input image can be captured using a wide-angle fisheye lens or another type of lens that introduces distortion. By rendering small blocks individually (e.g., 16x16 blocks or other sizes), the degree of distortion introduced when performing projective mapping from the input image to the output image can be reduced relative to rendering the entire output image as one large block. In some embodiments, the block size can be selected to maintain a balance between a threshold acceptable level of distortion and the amount of processing resources required to process a larger number of blocks, while maintaining a desired level of throughput and latency. For example, a smaller block size results in a larger number of blocks for the entire output image, which may require a greater number of parallel processing resources to render the output image with a desired latency. The desired balance between processing resources and output image distortion can be selected based on the computational capabilities of the processor system and the use case for the output image (i.e., desired throughput, resolution, and / or latency, among other possibilities). For example, for an embodiment in which an output image sequence is rendered to produce an output video in real time at 30 frames per second, a degree of parallel processing resources can be selected for a particular block size such that the latency in rendering the output image is less than 1 / 30 of a second. In other words, in some embodiments, the size of each of the plurality of pixel blocks can be selected to balance the degree of image distortion introduced by projection mapping and the amount of processing resources available to render the output image with an acceptable latency for a particular use case.

[0092] At 504 , for each pixel block of a plurality of pixel blocks of the output image, a mapping between a plurality of pixels along a perimeter of the respective pixel block and a plurality of corresponding pixels of each of one or more of the input images is determined.

[0093] In some embodiments, the plurality of pixels along the periphery of the corresponding pixel block are the four corner pixels of the corresponding pixel block. For example, the pixel block may be a rectangular pixel block of 4x4, 16x16, or another size, and the mapping between the plurality of pixels along the periphery of the corresponding pixel block and the plurality of corresponding pixels of each of the one or more input images in the input image may be performed for the four corners of the square pixel block of the output image. Alternatively, in other embodiments, pixel blocks of other shapes may be used, and the peripheral pixels may be the corners of the corresponding shapes. For example, the output image may be divided into tiled triangular blocks, and the peripheral pixels may be the three corners of each triangular pixel block. Other tile shapes, such as pentagons, hexagons, etc., may also be used as desired. The above reference may be used to represent the image. Figure 4For example, the projection surface may be used to determine a mapping between four corner pixels of a pixel block of an output image and four corresponding pixels in one or more input images.

[0094] At 506, for each of the one or more input images, a projection mapping is performed on pixels located within a perimeter of a corresponding pixel of the input image to a corresponding pixel in the output image. The projection mapping can be based at least in part on a mapping between a plurality of pixels along a perimeter of the corresponding pixel block and a plurality of corresponding pixels of the one or more input images. For example, once the four corner pixels of the pixel block of the output image are mapped to the four corresponding pixels of the one or more input pixels, the mapping between pixels located within the four corners of the output image (e.g., these pixels located within the perimeter pixels can be referred to as "interior pixels" of the pixel block of the output image) and pixels of the one or more input images can be performed by linear extrapolation of the mapping of the corner pixels to the interior pixels of the pixel block output image.

[0095] The mapping of the peripheral pixels of the output image to one or more input images may be more computationally intensive than the projection mapping of the interior pixels. Advantageously, the mapping of the peripheral pixels is performed only for a relatively small number of pixels (e.g., four in the specific example described herein), while the much faster projection mapping of the interior pixels is performed for a larger number of pixels (e.g., 256-4=252 pixels for a 16x16 pixel block), thereby reducing the overall processing load of the rendering process.

[0096] Additional Notes

[0097] The following numbered paragraphs provide additional details and explanations about the embodiments described herein. For ease of reference, the sections and subsections of the following paragraphs are numbered according to a hierarchical numbering system. Specifically, section headings are marked with a single digit (e.g., "2"), subsections are marked with two digits (e.g., 2.3), and subsections of subsections are marked with three digits (e.g., 2.3.1).

[0098] 1. Overview

[0099] The following sections provide additional details related to the vehicle ADAS embodiment (Section 2); image processing algorithms (Section 3), including calibration and view generation functions (Section 3.1) and rendering functions (Section 3.2); software (Section 4), including parameters and configuration files (Section 4.1) and software components (Section 4.2); parallel processing hardware system design (Section 5); and chip design (Section 6).

[0100] 2. Vehicle ADAS Application

[0101] In some embodiments, a system including multiple cameras can be mounted on a vehicle to generate any view of the world around the vehicle. In some embodiments, four cameras with wide-angle field of view (FOV) optics can be positioned on each side of the vehicle and networked to a computing platform. The computing platform can capture images from the cameras and synthesize a single display image that is a combined view from a specified viewport (e.g., from a position around the vehicle, such as above, looking down at the vehicle). After the four images have been corrected for lens distortion and stitched together, a digital visual interface (DVI)-based display or other type of display can present the synthesized view. A final perspective transform can be applied to specify the virtual viewport and / or perspective to be presented in the display.

[0102] A static calibration process can be used to generate the various spatial and intensity corrections required to pre-process individual camera images.

[0103] The operator may be able to control the selection of the viewport using a tablet computer or other device with a simple user interface.Any world view image may be displayed directly from a monitor connected (e.g., wired or wirelessly) to the computing platform.

[0104] 3. Image processing algorithms

[0105] In some embodiments, any world view system includes four cameras with fisheye lenses (~180 degree FOV) to capture image samples around the car. In the horizontal plane, they can be oriented every 90 degrees, and each camera can have a 180-degree field of view, providing a lot of image overlap between adjacent cameras. Along the vertical axis, the cameras can be oriented at a 45-degree angle downward to provide full coverage and an all-around view of the ground around the car.

[0106] Image processing algorithms can combine image samples from these multiple camera sources into a single output view that represents a specific view perspective. The following sections describe the functional steps of rendering the output view in more detail.

[0107] The image processing steps can be divided into two groups. The first set of steps is performed only once, after the cameras are mounted on the car. Using a calibration target with a known position, this first set of steps can be used to calibrate the position and orientation of each camera relative to a common world coordinate system. A block-by-block description of each view based on the four cameras can then be generated. This first set of functions can be used for calibration and view generation for any world view.

[0108] The second set of operations includes a real-time system for rendering the view perspective at video rate using the block-by-block view description calculated during calibration. Image samples are received from the camera, a demosaicing function can be applied to restore full-resolution color, and then warping can be performed to project the samples onto the focal plane of a virtual rectilinear camera representing the perspective. If part of the view perspective is filled with samples from more than one camera, the samples from the two best cameras can be blended. Photometric alignment algorithms can also be applied in real time to compensate for exposure differences between cameras, which may change as a vehicle or other object moves through or experiences variable lighting conditions.

[0109] The general approach to applying these functions in both categories is to use the basic principles of multiple view geometry to achieve the desired functionality. In some embodiments, the model can use open source libraries such as OpenCV to implement many functions for discovering and applying projective transformations of images.

[0110] Figure 6 A data flow diagram illustrating two sets of processing included in a model according to some embodiments is shown. The upper portion of the diagram is a first set of steps that can be performed offline once for each calibration set and view specification. An array of render block parameters can be generated for each desired view specification and provided to a second set of operations that can be performed for each input image set from the camera.

[0111] 3.1 Calibration and view generation functions

[0112] The calibration function can be performed once, after the camera is installed on the car. The view generation function can be performed once for each view specification, which can include the position and orientation of the virtual camera relative to the car. The output of these functions can be a set of parameters from which the defined views can be repeatedly generated using image samples from one or four (or more) cameras to form the video output. The calibration and view generation functions can be offline processes running on a standard computer.

[0113] 3.1.1 Fisheye Distortion Correction

[0114] The concept of a projective transformation of an image can be based on an underlying projective camera model, such as the basic pinhole camera model. In this model, rays extend from a central point through a rectilinear image plane to an object in the world. The object's position in the world can be represented by the point where the ray intersects the image plane.

[0115] The projective camera model provides a basis for understanding the mapping between fisheye lens images and corresponding rectilinear representations by drawing rays to pass through both the rectilinear image plane and the curved fisheye image surface, e.g. Figure 7(In the diagrams throughout this document, the image plane is drawn as a line in one dimension, representing the 2D Euclidean region of the image.) Rays from the central projection camera are drawn in two dimensions, representing the 3D space of homogeneous image coordinates and world objects.

[0116] There are many different kinds of fisheye lenses. Assume that the lens used in some embodiments follows the equirectangular projection model, also known as the equidistant model, and is represented by the following equation:

[0117] r F =f F θ (1)

[0118] Among them, r F is the distance from the principal axis to the image point in the fisheye image surface, θ is the angle between the ray and the principal axis, and f F Similar to focal length.

[0119] The corresponding equation for the pinhole camera model is:

[0120] r P =f P tanθ, (2)

[0121] It provides a mapping from a fisheye source image to its rectilinear representation through θ. The model can use this mapping whenever the source image involves a projective geometric transformation. An example of conformal fisheye distortion correction is Figure 8A and Figure 8B As shown in . Figure 8A and Figure 8B The top of the figure shows the source image (180° field of view) with conformal (fisheye) distortion, and the bottom shows the reprojection using the pinhole camera model (120° field of view). As used herein, a “source image” may be understood as being the same as the reference image. Figure 3-5 The input image is described synonymously.

[0122] 3.1.2 Camera Calibration

[0123] The camera calibration function can find the camera position and orientation relative to the world coordinate system. The input to this process can be Figure 13A and Figure 13B An image of a checkerboard pattern is shown in . The following steps can be performed.

[0124] 1. Identify the pixel locations of the chessboard corners in the undistorted (straight) input image. This can be done using the cv::findChessboardComers() function.

[0125] 2. The user provides an array of world coordinates corresponding to the chessboard corner positions.

[0126] 3. Rotation matrix Rsource and the translation vector T source Perform a numerical solution so that the world coordinates can be reprojected into the undistorted source image. This can be done using the cv::solvePnP() function.

[0127] 4. Report the mean reprojection error in pixels, which measures the success of the camera calibration process.

[0128] The reprojection equation can be described as:

[0129] X source =R source X world +T source (3)

[0130] x source =KX source (4)

[0131] in:

[0132] is the chessboard corner position in the world coordinate system,

[0133] X source is the same point in the source camera coordinate system,

[0134] K is the intrinsic camera matrix describing the focal length and principal point parameters of the source image plane, and x source =s(u,v,1) T is the source image pixel location.

[0135] The reprojection equation is Figure 9 The rotation matrix R is shown graphically in source Rotate the world coordinate system to the direction of the camera coordinate system, and translate the vector T source Move the world coordinate origin to the camera center, which can be the optical center of a pinhole camera. source and T source It is sufficient to describe the camera's position and orientation in the world.

[0136] Note that the pixel position x and the intrinsic camera calibration matrix K can be related to a straight line representation of the source image. The fisheye lens mapping function can then be used to determine the actual pixel position in the fisheye source image corresponding to the checkerboard corner.

[0137] The camera calibration function provides the rotation matrix R source and the translation vector T source , which are the external calibration parameters of the camera. Using these parameters, any point X in the world coordinate system can be world =(x,y,z) TProjected into the source image. As described in more detail below, this can provide the basis for creating new images based on any virtual viewpoint.

[0138] Once the cameras are attached to the car, these extrinsic calibration parameters can be fixed. The camera calibration process can be performed once to find the rotation matrix R for each camera. source and the translation vector T source These parameters can be used to generate any number of views around the car using the same set of source cameras, as long as the relative positions and orientations of the cameras do not change.

[0139] 3.1.3 Protective Transformation

[0140] In some embodiments, as described in Section 3.1.2, a projective transformation is used between the world coordinates and the calibration image in the reprojection equation. The position of the world object is known in the world coordinate system, with its z-axis oriented perpendicular to the ground. The projective transformation can include rotating the coordinate system to the orientation of the camera coordinate system and then adding a translation to move the origin to the center of the camera. The transformed 3D coordinates of the world object can then be interpreted as homogeneous coordinates in the image projection plane to derive the pixel position of the world object.

[0141] Creating a new view for some virtual camera position using image samples from a source image may involve two projective transformations, such as Figure 10 First, each pixel position in the view image can be projected to the position of the object in the world coordinate system. As used herein, a “view image” can be understood as being the same as the reference image. Figure 3-5 Second, this world point can be reprojected back to a pixel location in the source image.

[0142] The mapping from view image to source image can be described by the following sequence of steps:

[0143] 1. Order is a homogeneous representation of a pixel location in the view's image coordinate system. The specific point value can be chosen so that z is the actual depth of the world object providing the image at that pixel.

[0144] 2. Rotate the axis from the view coordinate system to the world coordinate system.

[0145] 3. By adding P view (the position of the view camera in the world coordinate system), shifting the origin from the view camera center to the world origin:

[0146]

[0147] 4. Rotate the axes from the world coordinate system to the image coordinate system.

[0148] 5. Transfer the origin from the world origin to the image camera center:

[0149] X source =R source X world +T source (6)

[0150] The combined equation shows where to get the sample value of a pixel in the view image from the source image:

[0151]

[0152] In projective geometry, X view represents the area extending from the camera center and extending in X view The image color at the pixel position where this ray passes through the image plane is X view The color of the world object at that location. The depth of the object can be defined as X view z-component of . One might expect that from the addition of T source and P view In the subsequent steps, X view Scaled to world units, since these values ​​are in world units. Note that the rotation matrix R source and are orthogonal matrices. They are unitless and do not change the length of the vectors they transform.

[0153] 3.1.4 Surface Model of World Objects

[0154] In some embodiments, determining the depth of an object from an arbitrary view camera center may be non-trivial.A three-dimensional model of the object in the world may be utilized to obtain the projection surface, which may not be practically or easily obtainable from the source image(s).

[0155] The view generation algorithm can utilize a hybrid surface model with two intersecting surfaces. It can be assumed that the world objects are on this surface. The function used to calculate the depth can involve calculating X view As the intersection points between rays representing image pixels with the surface model.

[0156] As a mathematical equation starting from the pixel position, we can first express that position in homogeneous notation,

[0157]

[0158] Then apply the inverse intrinsic camera matrix K:

[0159]

[0160] The scaling factor s can be found so that X view The z-component is the depth of the object from the viewing camera, so that subsequent operations can be scaled correctly for X view In fact, the intrinsic camera matrix K of the pinhole camera model can make s equal to the depth of the object from the viewing camera.

[0161] For embodiments where the world around the car is assumed to be ground level with z=0, the calculation of depth is shown in 11. In these embodiments, the view camera specification includes some height P above the ground and an altitude θ+φ view | z The camera position P at view , where θ is the angle between the view principal axis and the +z axis, and φ is the angle between the view principal axis and the image ray.

[0162] to X view The length of the ray is:

[0163]

[0164] And the depth is the component of this length along the view camera principle axis

[0165] depth=||X view ||cosφ (11)

[0166] A ground plane model works well for views around the car from above, but may not be ideal for views that include object points above the horizon. Vertical objects in the foreground may also appear distorted in a rendered view that assumes a horizontal ground plane surface model. As the view's altitude increases toward horizontal, more objects in the view may represent vertical surfaces, such as buildings, vegetation, other vehicles, and / or people, among other possibilities.

[0167] In some embodiments, a cylindrical projection surface model is introduced to provide more realistic rendering for views with non-vertical principle axes. This model assumes that object points are on a vertical surface that wraps around the car at a certain distance, such as Figure 12 As shown in . The main axis of the cylinder can pass through the center of the car in the z-direction.

[0168] Any world view system can assume how far the cylinder surface is from the car, but in many cases, the rendered view may not fully reflect how closely that distance matches the actual distance to the world objects. In some embodiments, the size of the cylinder can be related to the speed at which the car is moving. A stationary car can use a smaller distance to more accurately render adjacent obstacles, while a moving car may render a more realistic view by assuming a larger cylinder dimension.

[0169] Some embodiments operate by finding the intersection point of a ray extending from the virtual camera center toward the camera's principal axis in the (x, y) plane. The ray length may then be increased to account for any vertical component, yielding a hypothetical object point X on the cylindrical surface. view .

[0170] The shape of the cylinder boundary in the (x, y) plane can be an ellipse with the major axis oriented along the length of the car and the minor axis oriented along the width. As part of the view specification, the user can specify a single surface depth value that represents the distance from the edge of the car to the cylinder surface.

[0171] In a hybrid implementation, the view generation function can use both a ground plane model and a cylindrical surface model. The depth for a given ray from the view camera center can be calculated using both models, and the smaller value can be chosen. If the two models provide nearly equal depth values, the radius function can calculate a reduced value. The result might be a hypothetical world object surface shaped like a flat-bottomed bowl, vertical sides surrounding the car, and a smoothly rounded interface between them.

[0172] If the assumed cylindrical dimension is smaller than the distance to the real-world object, the blending model can cause interesting effects in the rendered image. For example, objects near a car may be rendered with the expected perspective, while more distant points may exhibit a fish-eye lens effect. The radius function can provide a smooth transition between these areas in the rendered image.

[0173] 3.1.5 Piecewise Linear Transformation from Source Image to View Image

[0174] The previous sections outlined the projective transformation that maps source image (i.e., input image) pixels to view image (i.e., output image) for an arbitrary viewpoint, which can be summarized as the following mapping function:

[0175] X source =R map X view +T map (12)

[0176] x=KX source (13)

[0177] According to the pinhole camera model, this may result in pixel position x=s(u,v,1) in the source image T As a final step, the nonlinear mapping described in Section 3.1.1 can be applied to obtain the pixel locations in the original fisheye source image.

[0178] While the mapping function can be evaluated for every pixel of the view image, this can result in a large amount of mapping information for each view. To address this issue and reduce rendering time and computational load, in some embodiments, the output view image can be divided into a grid of pixel blocks (e.g., 16x16 pixel blocks or other sizes or shapes), and the mapping function can be evaluated only for the corners of each block. Figure 13A and Figure 13B A collection of blocks from two fisheye source images is shown, corresponding to a 16x16 grid in the view image. As shown, white blocks are mapped to the view image, gray blocks are blended with blocks from the other source image and mapped to the view image, and black blocks are not used in the view image because a better source image can be used instead. The red trace indicates the centerline of the blended region between the two source images. A blended surface model can be seen, where points close to the car are assumed to be on the ground, and points further away are assumed to be on the surface of a cylinder surrounding the car.

[0179] The coefficients of the affine transformation can be calculated for each output block to warp the source image samples into the view image. While general projective transformations can be implemented as 3x3 matrix multiplication operations, affine transformations can be a special case with fewer degrees of freedom; the last row of the transformation matrix may not need to be used. Therefore, the mapping of pixels in each output block can be reduced to:

[0180]

[0181] Only six values ​​are used to describe the affine transformation of each output block, and the source image positions of the four corners of each view image block provide enough information to solve for the value a 11 ...a 23 .

[0182] When the functions that map view image positions to source images include corrections for fisheye lens distortion, they may be nonlinear. Although affine transformations can be linear and can be more restrictive than general projective transformations, it may be sufficient to render the output view because it can be applied to each 16x16 output block individually. In other words, the degree of lens distortion can increase with the size of the pixel blocks into which the output image is divided. A 16x16 pixel block size may present an ideal balance between introducing a sufficiently small degree of distortion and processing resources because smaller pixel block sizes translate into a larger computational load. This forms a piece-wise approximation to the underlying nonlinear mapping. Some embodiments may balance computation for rendering and storage requirements for block mapping information in a real-time rendering system.

[0183] 3.1.6 Mixing ratio calculation

[0184] The fisheye lens on the source camera can provide a 180° field of view. In some embodiments, there are four cameras oriented in four cardinal directions so that there can be significant overlap between images captured by adjacent cameras.

[0185] In some embodiments, the blending ratio calculation can be used to determine which source cameras contributed to each block in the view image. For a given block location in the view image, a distance metric can be calculated based on the distance from the corresponding source image block and the center of the fisheye image. If the distance metric calculated for two sources is more equal than the distance metric for adjacent blocks, then the view image block can be identified as being located on the centerline between the two sources. This criterion may have the effect of favoring source samples closest to the center of each image with the highest resolution.

[0186] In some embodiments, each source image may include an exclusion boundary to disqualify source samples that should not appear in the rendered view image. If a source image block is close to an exclusion boundary, its distance metric may be adjusted to move the centerline away from the exclusion boundary.

[0187] After the centerline block in the view image has been identified, neighboring blocks can be assigned blending ratio values ​​based on their distance from the centerline. When the view block is composed of contributing blocks from two source images, the source block closest to the center of its own image can be assigned a blending ratio between 0.5 and 1.0, while the other source block can be assigned a smaller complementary value (for example, the other source block can be assigned a weight value of one minus the value assigned to the closer block).

[0188] The blending ratio calculation may be the last step of the camera calibration and view generation functions.In some embodiments, the ratios for overlapping view image tiles may be included in the set of parameters defining each view for the rendering function.

[0189] 3.2 Rendering Function

[0190] In some embodiments, the rendering functions of any world view presentation system can be repeatedly executed to produce video output for a given view specification. In some embodiments, they can be implemented by a system such as HyperX TM The system can be implemented in real time on the Vision Integrated Display System (VIDS) platform.

[0191] In some embodiments, a model implementation of the rendering functionality can also be utilized. The model implementation can be run on a standard computer and can operate on a single framework to provide a platform for evaluating any world view algorithm. The functionality of the model is comparable to real-time HyperX TM The design is closely related, so the model can also provide test vectors to support HyperX TM Application development.

[0192] The input to the rendering function may include a set of parameters for each 16x16 block of the output image for a given viewpoint. Source images from four cameras may also be used as input to the rendering function.

[0193] 3.2.1 Demosaicing

[0194] In some embodiments, a color image from a single CCD digital camera can be captured through a Bayer filter, which can be a color filter array that forms a mosaic of red, green, and blue samples. A demosaicing function can convert the one sample value per pixel into three values ​​per pixel representing the complete red, green, and blue planes.

[0195] View HyperX in any world in real time TM In the system, a demosaicing function can be applied to a source image block corresponding to each 16x16 pixel block in the view image. The source image block can have a rectangular shape and its size can be smaller or larger than the size of the view image block produced after applying the affine transform. The rectangular input block can be referred to as a "bounding box" because it completely contains the source image samples for each view image block. Figure 11 For a grid of blocks shown in , the bounding box can be an orthogonal rectangle that completely contains each source image block.

[0196] In various embodiments, different demosaicing methods can be selected. If the bounding box is 1000 pixels or smaller, a conventional demosaicing algorithm can be applied. In some embodiments, two missing color samples can be interpolated at each pixel location.

[0197] For bounding boxes between 1000 and 4000 words, a downsampling demosaicing algorithm can be used. Each pixel sample can be calculated as a weighted average of neighboring samples of the corresponding color to complete filtering, downsampling, and interpolation in the filtering operation. The output bounding box can be scaled down by a factor of two in both the vertical and horizontal directions relative to the input bounding box. The affine transformation coefficients can be pre-compensated to account for the reduced bounding box.

[0198] The downsampling demosaicing method extends the range of input block sizes that can be processed by any world view in real time. The output of the demosaicing operation can be a block of source image samples large enough to render a corresponding 16x16 view image block.

[0199] 3.2.2 Warping and Block Averaging

[0200] In some embodiments, a warp function is employed to apply the affine transformation.The input may be a source image block with 3 samples per pixel from a demosaicing function, and the output may be a 16x16 view image block.

[0201] For each block pixel location, the corresponding pixel location within the bounding box can be calculated as:

[0202]

[0203] The coefficient a 11 ...a 23 Can be part of a set of parameters computed by a view generation function. Can be a real-valued position, not limited to integer values ​​for u and v, so that the red, green, and blue sample values ​​at that position can be interpolated. Bilinear interpolation can be used.

[0204] In addition to warping the input block into a view image block, the average red, green, and blue sample values ​​can be computed across the view image blocks. The block averages can be used by the photometric alignment function described in Section 3.2.3 below.

[0205] 3.2.3 Photometric Alignment

[0206] A photometric alignment system can be used to independently compensate for different cameras and automatically adjust exposure and aperture settings in response to varying lighting conditions facing the vehicle from different directions. When two source images contribute overlapping patches to the view image, the resulting blended area can appear more natural if their photometric properties have been adjusted to match each other.

[0207] Figures 13A-13B Demonstrates overlap in multiple input images. As shown, source image patches with mid-gray borders lie within the overlapping blended regions of the view image. Photometric alignment can help equalize the colors of the source images so that the seams disappear in the rendered view image.

[0208] In some embodiments, block average sample values ​​can be compared in the overlap region between the two source images. A global gain correction value can be calculated for each source image to optimally equalize the sample values ​​in the co-located blocks. Separate gain values ​​can be calculated for each red, green, and blue plane. Using block averages collected over the entire view image, up to 12 gain values ​​can be calculated for the four source images.

[0209] The gain values ​​calculated for the previous set of source images can be applied to all image samples in the current view image. Each sample value can simply be multiplied by the gain value of the corresponding color and source image.

[0210] 3.2.4 Mixing

[0211] In some embodiments, the final step in the rendering process is to blend overlapping view image blocks from different camera sources. Most view image blocks contain samples from only a single source image, and the blending function for these blocks may be a transfer function. For the remaining view image blocks, the red, green, and blue values ​​may be combined pixel by pixel as a weighted average of two co-located rendering blocks from the two image sources.

[0212] As described in Section 3.1.6, the weight values ​​used for blending are calculated in the view generation process. The sum of the two weight values ​​contributing to a single view image block can be 1.

[0213] The output of the blending function may be a set of 16x16 view image blocks for those block positions whose source image samples are present in the set of source cameras. Some block positions in the view image may not map to any source camera, and these blocks may not be rendered.

[0214] The output blocks can be composited into frame images in external memory. A separate task can be responsible for reading the view images line by line from the external memory for presentation via the DVI output of the VIDS platform.

[0215] 4. Software

[0216] In some embodiments, any world view system can be provided in the form of four software programs that work together to render a single image for a given viewpoint based on image samples from one to four source images. Each program can be built and run independently, but they can share a common set of configuration and parameter files that enable them to work together.

[0217] 4.1 Parameters and Configuration Files

[0218] The programs in the model software application suite can take a number of parameters that describe information about the source camera, the desired view, and each 16x16 block or tile in the rendered view image. All programs can obtain these parameters from parameter files; in some embodiments, command line options are not typically used.

[0219] 4.1.1BlockInfo

[0220] Block-level parameters are called BlockInfo and are the bridge between the view generation operations described in Section 3.1 and the rendering operations described in Section 3.2. These parameters can be included in JavaScript TM The JSON format is a lightweight data exchange format borrowed from common web application programming practices that can be read by humans but can also be easily ingested by software.

[0221] The example BlockInfo structure is in Figure 14 The parameters are shown in Table 1 below.

[0222] Table 1. BlockInfo parameters

[0223]

[0224] The BlockInfo structure can be used in a file that contains all the block parameter sets describing each block in a view image for a given view specification.

[0225] De-mosaiced models and real-time any-world view HyperX TM The system can be used as Figure 15 The BlockInfo parameter to a C or C++ construct formatted as shown in . This can be in a binary format that may not be human-readable but can be processed efficiently in software.

[0226] 4.1.2CamData

[0227] In some embodiments, a CamData configuration file may be available in JSON format. It may be used by view generation functionality and may not be required for rendering. The camera_calibrate routine may use the CamData file as input and output, while the view_generate routine may use it as input. An example CamData file is in Figure 16 The CamData file is shown in Figure 2 and consists of three parts, as shown in Table 2. As shown, the first part contains the minimum set of parameters required for camera calibration. The second part adds the parameters required for view generation, which are the result of the camera calibration process. The third part may be information that is only read by human readers and may not be used by any program. Note that the functions used by software to read CamData files can read most existing files, so any unknown parameters in the second and third parts can simply be omitted from the file.

[0228] Table 2. Parameters of CamData file

[0229]

[0230]

[0231] In some embodiments, the targetPosition parameter can be based on Figure 17The world coordinate system is shown in . The center of the car is not necessarily the origin of the xy axis, but the camera calibration operation can assume the axis direction as shown. Specifically, in some embodiments, the -y axis direction can extend forward from the front of the car.

[0232] 4.1.3ViewSpec

[0233] A ViewSpec describes the desired viewpoint of a virtual camera. "Generating" this view may involve generating a file of BlockInfo structures used to render the view specified in the ViewSpec. In some embodiments, the ViewSpec file may be used only in the view generation process and may not need to be used by the real-time rendering system.

[0234] Figure 18 An example ViewSpec file is shown in FIG, and includes the parameters described in Table 1. The angular parameters describing the view orientation are referenced to the front axle of the car, such as Figure 19 As shown in .

[0235] Table 3. Parameters of ViewSpec file

[0236]

[0237]

[0238] 4.2 Software Components

[0239] The model software for any world view system can include two libraries and a number of command-line applications. These libraries implement much of the functionality of the view generation and rendering algorithms. Applications can call library functions and add code to handle high-level operation sequences and file I / O.

[0240] 4.2.1OpenCY TM Library

[0241] OpenCV TM It is an open source library of functions that supports computer vision applications. It is available in packaged form on all standard platforms and provides a core set of primitive data types and functions. For example, all operations involving projective geometry are expressed in C++ as simple mathematical operations, even if they represent linear algebraic equations involving vectors and matrices. Some functions, such as the singular value decomposition (SVD) algorithm used to find the affine transformation of each view block, are very complex and OpenCV TM The use of may be advantageous for efficient implementation of the model. In some embodiments, any world view model uses OpenCV TM C++ API version 2.4.

[0242] 4.2.2ArthroCV Library

[0243] In some embodiments, the ArthroCV library contains many functions of the model algorithm. It provides a C++ application programming interface (API) so that applications can access more than OpenCV TM The more primitive functions in are more specific to common functions of any world view application. For example, the mapping of source image samples between an equirectangular (fisheye) camera model and a pinhole camera model (Section 3.1.1) can be implemented as ArthroCV TM C++ classes that are part of the API.

[0244] ArthroCV TM The library can be used as a single shared object library for Windows TM arthrocv.dll or for Linux TM libarthrocv.so is provided.

[0245] 4.2.3 Demosaicing

[0246] In some embodiments, a demosaicing routine can be a standalone program to convert images captured by a single CCD camera. The input to this function can be a raw binary image with a Bayer mosaic of color samples. As an example, if the raw binary image contains 10 bits per pixel, the output can be a 32-bit per pixel file with 10-bit red, green, and blue samples per pixel position.

[0247] In some embodiments, to convert a raw image file, a demosaicing program may be invoked as follows:

[0248] demosaic-image blockinfo_image_alignment.bin bayerInputFile.rawoutfile.raw

[0249] Among them, "blockinfo_image_alignment.bin" contains a single binary BlockInfo parameter set that describes the source image including the Bayer pattern alignment.

[0250] A shell script may be provided that wraps the demosaicer with additional open source utilities to convert to and from other standard image formats, such as Portable Grayscale (.pgm) and Portable Graphics Format (.png).

[0251] 4.2.4 Camera Calibration

[0252] According to some embodiments, the camera calibration function is described above in Section 3.1.2. The result of camera calibration may be a rotation matrix and a translation vector, which tell the position and orientation of the camera in the world coordinate system.

[0253] Camera calibration can be performed once for each camera after it is mounted on the car. Images can be captured by the camera with a checkerboard calibration target. The parameters of the camera and target can be specified in Section 1 of the CamData configuration file, as described in Section 4.1.2 and Table 2.

[0254] Subsequently, camera calibration can be called via:

[0255] camera_calibrate cam0.json

[0256] According to some embodiments, the program may rewrite the camera configuration file with the updated parameters in sections 2 and 3. In some embodiments, it may also display two images to verify correct operation. An example of the first image is shown in Figure 20 , which can be the original source image, with identification of the circular boundaries of the fisheye lens field of view. The software can automatically discover the center and radius of the circular image, which can be used to create an accurate mapping between the conformal (fisheye) camera model and the ideal pinhole camera model representation of the image data. The first verification image can also show the exclusion boundaries specified in the CamData configuration file.

[0257] An example of the second verification image is Figure 21 The diagram in Figure 1 shows the calibration target after the fisheye lens distortion has been corrected. As shown, the corners of the checkerboard image are automatically detected in the image. The calibration routine uses these corners to solve for the camera's position and orientation.

[0258] The camera_calibrate program can report the reprojection error, which is the average error in the checkerboard corner positions reprojected back into the world coordinate system of the source image. The units of the reprojection error can be pixels in the pinhole camera representation of the source image. In some embodiments, a reprojection error of about 1 pixel or less may be desirable.

[0259] Multiple cameras can be used to form a hybrid view image for any view. The view generation algorithm may depend on the cameras maintaining a fixed relative position and orientation to each other after calibration. Therefore, it may be desirable to capture calibration images for all cameras while the car is parked in one position on the ground.

[0260] 4.2.5 View Generation

[0261] In some embodiments, a single program contains all the view generation operations described in Sections 3.1.3-3.1.6. The input can be a ViewSpec configuration file (e.g., see 4.1.3) that provides the position, orientation, size, and field of view of the virtual camera. The output can be a file containing BlockInfo parameters set for each 16x16 tile in the rendered image. This file can be input to the real-time view rendering system.

[0262] A BlockInfo parameter set can reference source image data from as many cameras as are provided on the command line (e.g., up to four). For example, to create a file of BlockInfo parameters for a given view, the following command might be used:

[0263] view_generate top_down_view.json cam1_front.json cam2_rear.json cam3_left.json cam4_right.json

[0264] Among them, view0.json is the ViewSpec configuration file, and the other files on the command line are CamData configuration files. Any number of CamData configuration files can be specified (in this embodiment, up to four), and their order on the command line can be arbitrary.

[0265] In some embodiments, when view_generate runs, it shows the source image corresponding to each CamData profile, with an overlay of the tiles used in the output view image. Figure 13A -B shows an example of these annotated source images. As shown, light blocks are mapped to the view image, gray blocks are mixed with blocks from another source image and mapped to the view image, and faded black blocks are not used for the view image because a better source image can be used instead. A series of marks indicate Figure 13A and Figure 13B The blending area between the two source images.

[0266] 4.2.6 View Rendering

[0267] In some embodiments, the view_render program is used as a model implementation of the entire rendering process described in Section 3.2.

[0268] The BlockInfo parameter file produced by view_generate can be a complete block-by-block description of how to render a given view from multiple camera sources. The only additional data utilized on the command line can be the filename of the source image itself and its cameraId value.

[0269] An example command line is:

[0270] view_render blockinfo.json 1front_c.png 2rear_c.png 3left_c.png4right_c.png

[0271] The first argument is the BlockInfo parameter file generated by view_generate. The next arguments are the {camerald, sourceImage} pairs of all cameras used in the generated view.

[0272] The image file names may not need to be the same as the files provided for calibration, as this functional model may be used for systems that provide up to four source images by real-time capture from cameras running at video rate. However, it may be desirable for each image file to be associated with the same camera as the calibration images captured by the same camera.

[0273] Figure 22 is an example of a rendered view. The missing block in the center of the image is due to the car itself blocking the view of the ground from all four cameras. The rest of the image can be composed of data from three cameras, with blending transitions between them. Even though a fourth camera might be specified on the command line, its image samples might not be used in the rendered image if they are not within the field of view described by the ViewSpec configuration.

[0274] 5. Parallel Processor System Design

[0275] The following sections outline the design of the view rendering portion of any world view system and method, according to some embodiments. The design can be based on a HyperX 1000 controller that includes four GigE Vision camera inputs and a direct output for a DVI display. TM HyperX in Development Systems TM HX3100 processor, or it may be based on another type of parallel processing hardware system.

[0276] 5.1 Functional Analysis

[0277] Functional analysis can consider the computational operations of the view rendering algorithm, as described in Section 3.2 above. The number of basic operations can be estimated for each function, which can then provide an estimate of how many processor cycles can be utilized to execute the function. By knowing the cycle count of each function used to process a block of the view image, it is possible to know how many processor elements the system can utilize to run in real time to produce a view image at a specific resolution (e.g., 1440x900 or other resolution) at a specific frame rate (e.g., 30 or 60 frames per second).

[0278] According to some embodiments, the main computational steps of any world view rendering function are described in the following sections. According to the following set of example specifications, the block throughput requirements of each step in the process can be based on the target view image size and frame rate.

[0279] 5.1.1 Demosaicing

[0280] In some embodiments, two function cores may be used for the demosaicing algorithm (e.g., as described in Section 3.2.1): one for interpolating red and blue pixels at green locations in the Bayer mosaic pattern, and a second for interpolating green and red / blue pixels at blue / red locations.

[0281] Next, we can assume a block size for the demosaicing process. This can be the bounding box of the source image data used to render a single output 16x16 block. This analysis assumes a worst-case size of 1000 samples per block. Recall that if the bounding box size is between 1000 and 4000 samples, then an alternative downsampling demosaicing algorithm can be used.

[0282] 5.1.2 Distortion

[0283] As described in Section 3.2.2, the warp function can be calculated from a rectangular block of source image samples for a 16x16 view block of 256 pixels. The two parts of the function can include applying an affine transform to obtain a (non-integer) position in the block of input samples, and interpolating the input samples based on that position. Block average red, green, and blue values ​​can also be calculated in the warp for use by the photometric alignment algorithm.

[0284] 5.1.3 Photometric Alignment

[0285] The photometric alignment algorithm (Section 3.2.3) can be implemented in two cells, one for gain estimation and the other for gain correction, which applies the gain value to adjust the color sample values. The gain estimation cell calculates the sum of squared differences and applies a threshold function to reject poorly matched blocks.

[0286] At the end of each frame, the gain estimation cell also solves a system of equations involving the stored sums of squared differences.

[0287] 5.1.4 Mixing

[0288] A blended view block can be a view block with two contributing sample values ​​from the source image. In this example embodiment, it is assumed that 20% of the view image blocks are blended, and the remaining 80% are not blended. Blending calculations can only be applied to blended view blocks. Other blocks can simply be copied from a single input to the output.

[0289] 5.2 Architecture Considerations

[0290] This section lists various architectural considerations that can impose constraints on system design, including hardware platform details and high-level functional specifications. In some embodiments, any world-view system can perform its functions and meet the requirements described by these considerations. These considerations, along with the functional analysis in the previous section, can be important factors in determining the performance of a system design.

[0291] 5.2.1 High-level functional specifications

[0292] High-level functional specifications that are considered as architectures may be those that determine the target input / output and computational throughput of the system.In some embodiments, the following specifications may be established in any world view system requirements document.

[0293] The demonstration system can provide a video monitor that displays the user-selected vehicle view. The display has a nominal resolution of 1440x900 and can use a DVI signal interface.

[0294] The demonstration system can capture images from the camera at a rate of at least 30 fps and process frames to the display at the same rate. The actual refresh rate of the display may be higher, but the content processed by the system can be updated at a rate of at least 30 fps.

[0295] 5.2.2 Hardware Platform

[0296] In some embodiments, the computing platform for any world view application may be a HyperX processor equipped with three chips labeled Chip 1, Chip 2, and Chip 3. TM HyperX processors TM development system. Chips 1 and 2 can each be provided on a board with a Gigabit Ethernet interface, including HyperX TM Software driver in cell form to send and receive UDP / IP packets, such as Figure 23The software I / O driver unit can support the GigE Vision image transmission protocol, and each chip can receive video from two cameras simultaneously.

[0297] Figure 19 .Chip 1 and 2 resources

[0298] In some embodiments, chips 1 and 2 may each have these additional capabilities and resources:

[0299] 1. Four external memories, each with at least 128MB of storage

[0300] 2. Two LVDS I / O interfaces to the left neighbor, with a native bandwidth of 125MW / s

[0301] 3. Two LVDS I / O interfaces to the right neighbor, with an original bandwidth of 125MW / s

[0302] 4. Set the core clock to 500MHz

[0303] 5. 100 processor elements and 121 data memory routers, fewer than the processor elements and data memory routers used for the Gigabit Ethernet driver cell.

[0304] Figure 24 The third chip shown in FIG is provided on a board with a DVI interface for video output. A software video output driver unit may be provided to allow the HyperX 1000 series to be used from the HyperX 1000 series running on chip 3. TM Applications connect directly to standard DVI computer monitors via the on-board hardware line driver.

[0305] In some embodiments, chip 3 may have the following additional capabilities and resources:

[0306] 1. Three available external memories, each with at least 128MB of storage

[0307] 2. Two LVDS I / O interfaces to the right neighbor, with a native bandwidth of 125MW / s

[0308] 3. Core clock set to 425MHz (this rate is used for 1440x 900x 60fps DVI output)

[0309] 4.100 processor elements and 121 data memory routers, less than the processor elements and data memory routers used for the DVI output driver cell

[0310] 6. Chip Design

[0311] The previous sections describe the system design for any world view system that renders view images in real time at video rates. Section 3.2 describes the algorithms that the rendering system can perform, and Section 4.2.6 provides a software model for them. Section 5.2.1 describes the high-level performance specifications, and Section 5.2.2 describes the target platform.

[0312] In various embodiments, there may be more than one solution for the system design of this application, and the implementation details of different solutions may vary. The specific solutions described in this section are intended to meet the specification requirements and support the general engineering goals of cell reuse and design maintainability.

[0313] 6.1 Three-chip Design

[0314] In some embodiments, an architectural constraint imposed by the development platform might be receiving video from two cameras on each of two different chips. This can make it convenient to divide the main computational work of rendering processing evenly between the two chips. In some embodiments, this can be achieved through advanced workload partitioning, such as Figure 25 As shown in the figure, from right to left, video can be received and processed in Chip 1 and Chip 2, labeled HX1 and HX2. The processing on each chip can take advantage of access to video samples from all four cameras, so the two chips can exchange video data that they do not directly receive.

[0315] The third chip, labeled HX3, can receive the partially processed view blocks from the first two chips and can complete the rendering process and send the video frames to the DVI display.

[0316] The internal designs of the three chips are described in more detail in the following sections.

[0317] 6.2 Chip 1 and 2 Design

[0318] Figure 26 The diagram illustrates the data flow design for chips 1 and 2 according to some embodiments. As shown, the GigE Vision input for camera data is located in the upper right corner of the diagram. The processing network may include one or more of the following elements:

[0319] 1. Two parallel paths for unpacking and writing input images to DDR external memory

[0320] 2. MemSvr3 and MemSvrExt3 cells allow four camera streams to be stored in external memory, two from the local Ethernet interface and two from another chip

[0321] 3.BlockInfo parameters are stored in different DDR external memories and read by the generation cell

[0322] 4. InputReader reads the bounding box as specified in each BlockInfo parameter of each block

[0323] 5. The demosaicing group includes many demosaicing cells

[0324] 6. The twister group includes multiple twister cells

[0325] 7. Include FIFOs on some communication channels to absorb variations in block processing rates due to BlockInfos of different sizes.

[0326] 8. The output from chip 1 and chip 2 can be a stream of view image blocks sent to chip 3.

[0327] 6.2.1 Demosaicing Group

[0328] The demosaic_group cell can be a single cell with two inputs and one output. Within the demosaic group, there are many demosaic cells and other PEs dedicated to workload distribution. According to some embodiments, the internal structure of the demosaic_group cell is Figure 27 A twister group can have a similar structure except that it has only a single input.

[0329] 6.3 Chip 3 Design

[0330] Chip 3 receives a stream of view blocks from Chip 1 and Chip 2, such as Figure 28 They can be merged into a single stream so that all view blocks in the output image can be considered together for photometric alignment. Functional analysis of cells on chip 3 shows that more than one or two PEs per function may not be needed (see Sections 5.1.3 and 5.1.4).

[0331] The output of the photometric alignment composite cell can be a stream of view blocks in their final form. These blocks can be written to a frame buffer in external memory. The completed rendered frame can then be read line by line from the external memory for display via the DVI output cell.

[0332] 6.3.1 Photometric Alignment of Composite Cells

[0333] In some embodiments, all image processing functions in chip 3 can be surrounded by a photometric alignment composite cell, such as Figure 29The functionality may include gain estimation, gain correction, and mixing as described in Sections 3.2.3, 3.2.4, 5.1.3, and 5.1.4.

[0334] 6.4 System Design Goals

[0335] There may be more than one system design that can produce a rendered view image for any world view system. The specific designs described in the previous figures and paragraphs were chosen to meet a number of general design goals that are important for a HyperX-based system. TM However, these specific design parameters are intended to be illustrative only, and it should be understood that it is within the scope of the present disclosure to implement the embodiments described herein using other design parameters and systems.

[0336] 6.4.1 Localization

[0337] The different functions performed in the previous sections can be divided into different cells. Some cells, such as the external memory interface cell, can be used primarily for data movement rather than computation. Locality can be an important goal of the chosen design decomposition (e.g., the partitioning of an algorithm among many cells). Locality can be thought of as meaning that each cell can perform its function without knowing what functions are being performed by other cells in the design. A cell can receive unique information on its inputs that it can use to perform its task, and it can send only work products to be used by downstream cells to perform their tasks.

[0338] An example of localization can be observed in any world view system, as many cells in the design are unaware of which frame their workload belongs to when processing a view block. The functionality of each block may be identical. Other cells may be responsible for tracking frame boundaries and updating the frame pointer in external memory as needed.

[0339] The interfaces of the cells in a system can provide evidence of good localization in the design. If the interface of each cell is as simple as possible, then localization is likely good. Compared to some other design decompositions, simple interfaces may have a small number of input and output ports, and the input and output data size of each workload is minimized.

[0340] Focusing on localization in a design can result in simple cells that are often reused in a design and other designs. According to various embodiments, the demosaicer and distorter groups are examples of cells that can be reused within a design, and the external memory interface is an example of a cell that can be reused in many designs.

[0341] 6.4.2 Feedforward

[0342] In some embodiments, a feed-forward design may be utilized, which can be understood to mean that each cell in the design receives data from an upstream cell, performs its function on the data, and then sends the result to a downstream cell. The design may not have explicit feedback communication, such as a cell sending data back to an upstream cell in the design.

[0343] A purely feedforward design can have a simple model for flow control and performance. The desired throughput for any worldview system might be 30 frames per second, which translates into a specific block processing rate. If each cell in the design can perform its function at the desired rate, then the overall design can also meet the performance target, as long as there are no feedback paths in the design. The flow control model can be localized within each individual cell; each cell can simply rely on its upstream neighbors to provide it with workload at the desired rate and on its downstream neighbors to accept its products at the desired rate.

[0344] 6.4.3 Distributed Control

[0345] In some embodiments, the principle of distributed control is applied to HyperX-based TM The parallel processing application of the technology to scale to arbitrary scale computations may be advantageous. Distributed control can be thought of as meaning that a cell receives its instructions only from its upstream neighbors and only sends instructions to its downstream neighbors. In some embodiments, this may be an aspect of localization.

[0346] The opposite of distributed control is global control, where there might be a controller cell on one side sending instructions to various other cells in the design via a star or other type of communication topology. A problem with distributed control can be synchronization. For example, when should the global control cell send its instructions? One part of the design might be processing a different location in the video frame than another due to the processing pipeline between them. With distributed control, timing skew within the design may be less significant and may not need to be considered.

[0347] An example of distributed control in any world view system can be seen in two sets of frame buffers, one for source images and the other for display images. The decision to advance from one frame to the next can be handled locally in the writer and reader cells. Advantageously, in some embodiments, this can be achieved without coordinating frame synchronization between the source image buffer and the display image buffer.

[0348] MPS System Overview

[0349] The following paragraphs provide additional details on hardware configurations for a multi-processor system (MPS), such as a HyperX TM architecture or another type of MPS.

[0350] Various embodiments of multiprocessor systems (MPSs) and associated methods are described. A multiprocessor system (MPS) can be defined as a system that includes multiple processing elements (PEs). The MPS can have multiple memories distributed among the PEs, or alternatively can have a single shared memory. As used herein, the term "processing element" refers to a processor or CPU (central processing unit), a microprocessor, or a processor core. Although an MPS can include any of two or more PEs, it is noted that some MPSs can include many more PEs than traditional computer systems, which typically only include one general-purpose processor (GPP) or a few GPPs. For example, some MPSs can include 4, 8, 16, 32, or 64 PEs (other examples include, for example, tens, hundreds, or even thousands of PEs). In some embodiments, PEs suitable for large MPSs can be more energy efficient than general-purpose processors used by traditional computer systems because they have special construction for low power consumption.

[0351] The MPS may also include an interconnection network (IN) that interconnects PEs and / or memories. PEs and memories may be interconnected in one, two, three, or more dimensions, including circular dimensions (e.g., loops or rings). A higher-dimensional MPS may be mapped onto a smaller-dimensional manufacturing medium. For example, an MPS having a four-dimensional (4D) hypercube shape may be mapped onto a 3D stack of silicon integrated circuit (IC) chips, or onto a single 2D chip, or even onto a 1D line of computing units. Similarly, a low-dimensional MPS may be mapped onto a higher-dimensional medium. For example, a 1D line of computing units may be arranged in a serpentine pattern on a 2D plane of an IC chip, or coiled into a 3D chip stack. An MPS may include an interspersed arrangement of multiple types of computing units and processors and memories. Broadly speaking, an MPS also includes a hierarchical or nested arrangement of MPSs, particularly an MPS consisting of interconnected IC chips, where the IC chip contains one or more MPSs, which may also have a deeper hierarchy.

[0352] As used herein, the term MPS encompasses both relatively homogeneous collections of processors and heterogeneous collections of general-purpose and specialized processors integrated on so-called "platform IC" chips. Platform IC chips can contain from a few to many processors, typically interconnected with shared memory and possibly an on-chip network. There may or may not be differences between MPS and "platform IC" chips. However, "platform IC" chips can be marketed to address specific technical requirements in specific vertical markets.

[0353] Generally speaking, memory for an MPS can be organized in a hierarchy, with fast memory at the top and slower, higher-capacity memory at each step down the hierarchy. In an MPS, support memory (SM) at the top of the hierarchy can be located near each PE. Each support memory can be dedicated to holding only instructions or only data. The support memory for a particular PE can be private to that PE or shared with other PEs.

[0354] Further down the memory hierarchy, there may be larger shared memories, such as semiconductor synchronous dynamic random access memory (SDRAM), which has a bit capacity many times greater than the capacity of the support memory adjacent to each PE. The SDRAM may be located on one or more IC chips separate from the PE and support memory for dedicated manufacturing. Further down the memory hierarchy, there may be other types of memory, such as flash memory, magnetic disks, and optical disks.

[0355] The MPS can be programmed with software programs to perform specific functions. Each function can be executed by one or more PEs within the MPS. Typically, multiple programs can be executed concurrently on the MPS. By employing parallel processing techniques, programs can execute together and communicate with each other to perform more complex functions or execute simpler functions more quickly. This coordination between PEs is referred to herein as collaborative processing.

[0356] The MPS can execute an application or program quickly enough so that it can accept input data and commands faster than the associated data source, and the commands can provide them and provide results with a latency low enough to be negligible. Such applications are said to operate in real time without delay, or are referred to as "real-time applications." The associated input data (or commands) can be referred to as "real-time data" (or "real-time" commands). For example, the MPS can receive real-time data via an input signal. One or more of the applications, programs, or functions can process the input signal and, possibly based on one or more programs, generate an output signal with modified or additional real-time data.

[0357] Figure 30 -MPS Block Diagram and Overview

[0358] Figure 301 is a block diagram illustrating one embodiment of a multiprocessor system (MPS). In the illustrated embodiment, the MPS 10 includes a plurality of processor elements (PEs) and a plurality of data memory routers (DMRs), which may also be referred to as dynamically configurable communicators or dynamically configurable communication elements, coupled to transfer data and instructions to each other. As used herein, a PE may also be referred to as a PE node, and a DMR may also be referred to as a DMR node.

[0359] The processing system (MPS) 10 can be used in any of the various systems and applications currently using GPMCs, DSPs, FPGAs, or ASICs. Thus, for example, the processing system 10 can be used in any of various types of computer systems or other devices requiring computation. In one contemplated embodiment, the processing system 10 is used as a signal processing device in a digital video display system.

[0360] In one embodiment, a PE may include one or more arithmetic logic units (ALUs) configured to manipulate data, one or more instruction processing units (IPUs) configured to control the ALUs, one or more memories configured to hold instructions or data, and various multiplexers and decoders. Such an embodiment may include multiple ports ("processor ports"), some of which may be configured to connect to DMRs, while others may be configured to connect to other PEs.

[0361] In one embodiment, a DMR may include one or more random access memories (RAMs) configured to hold data and instructions, a configurable controller, a network switch such as a crossbar switch, registers, and multiplexers. Such an embodiment may include multiple ports, some of which may be configured to connect to a PE (referred to herein as a PE-type port), and other ports may be configured to connect to a DMR (referred to herein as a DMR-type port). Note that for any given port, whether configured to connect to or from a DMR or PE, the amount of data that can be transmitted through the given port in a particular clock cycle may vary in various embodiments. For example, in one embodiment, a given port may be configured to transmit one data word per clock cycle, while in another embodiment, a given port may be configured to transmit multiple data words per clock cycle. In yet another embodiment, a given port may employ techniques such as time division multiplexing to transmit one data word over multiple clock cycles, thereby reducing the number of physical connections comprising the port.

[0362] In one embodiment of the MPS 10, each PE may include a small local memory reserved for instructions and may include very little local data storage. In such an embodiment, the DMRs adjacent to each PE may be configured to provide operands to a given PE. In a particular embodiment, for many PE instructions, a given PE may read operands from an adjacent DMR, perform an ALU operation, and store the ALU result to a given adjacent DMR in a single clock cycle. Thus, an ALU result from one PE may be available to several other PEs within a single clock cycle immediately following execution. Generating results in this manner allows for close coordination or "tight coupling" of execution of adjacent PEs.

[0363] As used herein, from the perspective of a given DMR or PE, an adjacent DMR or PE refers to a DMR or PE that can be accessed from the given DMR or PE within a specific time delay. In some embodiments, the time delay defining the degree of neighbor relationship can vary, for example, depending on factors such as clock speed. In addition, in some embodiments, multiple degrees of neighbor can be defined, which can correspond to different access delays. For example, in one embodiment, a "nearest neighbor" can be defined as a device that can provide data during the same clock cycle that the data is requested, a "nearest neighbor" can be defined as a device that can provide data within one clock cycle after the data is requested, and so on. In other embodiments, it is contemplated that other metrics can be used to quantify neighbor relationships.

[0364] In a given MPS embodiment, some DMRs and PEs may be logically adjacent to other DMRs and PEs. As used herein, "logically adjacent" refers to a relationship between two devices, such as a DMR and another DMR, or a DMR and a PE, such that one or more ports of one device are directly connected to corresponding ports of the other device without passing through an intervening DMR or PE. In addition, in a given MPS embodiment, some DMRs and PEs may be physically adjacent to other DMRs and PEs. As used herein, "physically adjacent" refers to a relationship between two devices (such as a DMR and another DMR, or a DMR and a PE) such that no other DMR or PE is physically located between the two devices.

[0365] In some MPS embodiments, logically and / or physically adjacent devices, such as DMRs and PEs, are also considered adjacent or neighboring devices. However, it should be noted that, in some embodiments, logical and / or physical proximity between given devices does not necessarily imply a required adjacency relationship or a specific degree of adjacency between the given devices. For example, in one embodiment, a DMR may be directly connected to another DMR that is significantly distant. Such a pair may be logically adjacent but not physically adjacent, and the signal propagation time from one DMR to the other may be too great to meet the neighbor's latency requirements. Similarly, in one embodiment, a DMR may be physically adjacent to another DMR but not directly connected to it, and therefore not logically adjacent to it. Access from one DMR to another may traverse one or more intermediate nodes, and the resulting transmission delay may be too great to meet the neighbor's latency requirements.

[0366] Depending on the technology and implementation of a given embodiment of MPS 10, the number of ports for a particular number of DMRs and the size of the DMR memory can be balanced against the overall desired execution speed and size of the DMR. For example, a DMR embodiment may include four PE-type ports, four DMR-type ports, and 4K words of memory. Such a DMR embodiment may be configured to provide a direct memory access (DMA) mechanism. The DMA mechanism may allow a given DMR to efficiently copy data to or from other DMRs or to or from locations external to MPS 10 while a PE is computing a result.

[0367] In one embodiment of the MPS 10, data and instructions can be transferred between DMRs in one of several different ways. A serial bus can be provided to all memories within the MPS 10; such a bus can be used to initialize the MPS 10 from external memory or to support testing of MPS data structures. For short-distance transfers, a given PE can be programmed to directly move data to or from its neighboring DMRs. To transfer data or instructions over longer distances, communication paths can be dynamically created and destroyed within the network of DMRs.

[0368] For this longer distance data transmission, the interconnected DMR network within the MPS 10 can form a switch routing fabric (SRF) for the communication paths. In such an embodiment, there may be at least two methods for managing the communication paths in the SRF. The first method is through global programming, where the path can be selected by software control (e.g., by a human programmer or a compiler with routing capabilities), and instructions can be encoded into the DMR configuration controller to appropriately program the crossbar switch. To create a path, each DMR along the path can be explicitly programmed with a specific routing function. In a dynamic environment where paths are frequently created and destroyed, a large amount of crossbar configuration code may be required, the storage of which may in turn consume potentially limited DMR RAM resources.

[0369] A second approach to managing communication pathways is known as "wormhole routing." To implement wormhole routing, each DMR can include a set of steering functions and a mechanism to stop and restart the progression of a sequence of words (called messages) through the SRF. Because the steering functions can be shared and reused by all communication pathways, the amount of configuration code that can occupy DMR RAM can be much smaller than with the global programming approach described above. With the wormhole routing approach, software control can still be used to select the specific links to be used by a pathway, but the processes of pathway creation (also referred to herein as setup) and destruction / link release (also referred to herein as "teardown") can be implemented in hardware with minimal software intervention.

[0370] To prevent potential loss of data words along the path, embodiments of MPS 10 can implement flow control between receivers and transmitters along the path. Flow control refers to a mechanism that can stop a transmitter when its corresponding receiver is no longer receiving data, and restart it when its corresponding receiver is ready to receive data. Because stopping and restarting the flow of data along a path shares many similarities with stopping and restarting the progress of messages in wormhole routing, the two can be combined into an integrated solution.

[0371] In one embodiment, the MPS 10 may include a plurality of PEs and DMRs connected together in a uniform array, where the PEs may be identical and the DMRs may be identical. In a uniform array, most of the PEs may be identical, and each of the majority of the PEs may have the same number of connections as the DMRs. Furthermore, in a uniform array, most of the DMRs may be identical, and each of the majority of the DMRs may have the same number of connections as the other DMRs and PEs. In one MPS embodiment, the PEs and DMRs may be dispersed in a substantially homogeneous manner. As used herein, a substantially homogeneous dispersion refers to an arrangement in which the ratio of PEs to DMRs is consistent across a majority of a sub-area of ​​the array.

[0372] A uniform array arranged in a substantially homogeneous manner can have certain advantageous properties, such as providing a predictable interconnection pattern and enabling software modules to be reused across the array. In one embodiment, a uniform array can enable the design and testing of a small number of instances of PEs and DMRs. The system can then be assembled by manufacturing a unit comprising DMRs and PEs and then repeating or "tiling" this unit multiple times. This approach can reduce design and testing costs by reusing common system elements.

[0373] It should also be noted that the configurable nature of the PE and DMRs can allow a wide variety of non-uniform behaviors to be programmed to occur on a physically uniform array. However, in alternative embodiments, the MPS 10 can also be formed with non-uniform DMR and PE cells, which can be connected in a regular or irregular array, or even in a random manner. In one embodiment, the PE and DMR interconnects can be implemented as circuit traces on, for example, an integrated circuit (IC), a ceramic substrate, or a printed circuit board (PCB). However, in alternative embodiments, such interconnects can be any of a variety of micro-communication links, such as, for example, waveguides for electromagnetic energy (i.e., radio or optical energy), wireless (i.e., unguided) energy, particles (such as electron beams), or potential energy on molecules.

[0374] MPS 10 can be implemented on a single integrated circuit. In one embodiment, multiple MPS integrated circuits can be combined to create a larger system. A given embodiment of MPS 10 can be implemented using silicon integrated circuit (Si-IC) technology and can employ various features to take into account the specific characteristics of such technology. For example, the circuits on a Si-IC chip can be confined to a thin plane. Accordingly, a given embodiment of MPS 10 can employ a two-dimensional array of PEs and DMRs, such as Figure 30 However, alternative MPS embodiments are contemplated that include different arrangements of PEs and DMRs.

[0375] Furthermore, the available wiring density on a Si-IC chip may be much higher than the wiring density between such chips, and each chip may have a perimeter of dedicated input / output (I / O) circuitry to interface on-chip and off-chip signals. Accordingly, a given embodiment of the MPS 10 may employ a slightly non-uniform array consisting of a uniform array of PEs and DMRs in the chip core and modified PE / DMR cells along the chip perimeter. However, alternative MPS embodiments are contemplated that include different arrangements and combinations of uniform and modified PE / DMR cells.

[0376] Furthermore, the computational operations performed by Si-IC circuits may generate heat, which can be removed by the IC package. The increased IC package size may require additional space, and the interconnections through and around the IC package may introduce delays proportional to the path length. Therefore, as described above, very large MPSs can be constructed by interconnecting multiple chips. The programming of such multi-chip MPS embodiments can take into account that inter-chip signal delays are much longer than intra-chip delays.

[0377] In a given Si-IC MPS 10 embodiment, the maximum number of PEs and DMRs that can be implemented on a single chip can be determined by the miniaturization possible with a given Si-IC technology and the complexity of each PE and DMR. In such an MPS embodiment, the circuit complexity of the PEs and DMRs can be minimized while achieving a target level of computational throughput. Such minimized PEs and DMRs may be referred to herein as streamlined. In one MPS 10 embodiment, the target level of PE throughput may be comparable to the target level of the arithmetic execution unit of a best-in-class digital signal processor (DSP) manufactured with the same Si-IC technology. However, other MPS embodiments are contemplated in which alternative references for target PE throughput may be used.

[0378] In some embodiments, the MPS 10 can adopt the best features of DSP and FPGA architectures. Like a DSP, the MPS 10 can be a programmable chip with multiple processing units and on-chip memory. However, relative to the DSP, the MPS processing units can be simplified, there can be more processing units, and they can be interconnected in novel ways to maximize the bandwidth of data movement between them and data movement on and off the chip. Having more processing units than a DSP can allow the MPS 10 to perform more multiplication operations per unit time, and the simplified processing units can minimize energy consumption. Many DSPs with internal parallelism may be bus-oriented architectures. In some embodiments, the MPS 10 may not include a bus, but may include adjacent shared local memory embedded in the SRF, such as in the DMR, which can provide a much higher total bandwidth than a bus-oriented architecture.

[0379] Compared to FPGA approaches, some MPS embodiments may have a coarser granularity. For example, in one MPS embodiment, if the operation is performed using data that is a multiple of the natural word length, then the operation can have a natural word length (e.g., 16 bits) and the computation can be most efficient. In some MPS embodiments, the PEs and DMRs may be denser than equivalent structures implemented in an FPGA, which may result in shorter average wiring lengths, lower wiring capacitance, and less energy consumption. Compared to FPGA implementations, in some MPS embodiments, each ALU in the MPS may be part of a processor (i.e., a PE), which may facilitate the fetching of operands and writing results back to surrounding fast memory in the DMR. Timing and clock skew issues for ALU, fetch, and writeback operations can be addressed once during IC chip design, rather than having to be re-addressed for each new application as in a typical FPGA implementation.

[0380] MPS topology and communication

[0381] Figure 30 The MPS 10 shown in FIG can provide sufficient connectivity to fast memory for PEs by spreading DMRs among the PEs, as shown. Such an arrangement can reduce the time required for a given PE to access memory in the DMRs relative to a separate (i.e., non-spread) arrangement and can be referred to herein as a spread mesh arrangement. Figure 3 In the embodiment of the present invention, the ratio of PE to DMR is approximately 1: 1. However, other MPS embodiments are contemplated that may include different ratios of PE to DMR.

[0382] exist Figure 3 The connections between the DMR and the PE are not explicitly shown in FIG, as there may be many possible connection schemes using different types and numbers of connections.

[0383] Figure 31 -MPS connection solution

[0384] Figure 31 is a block diagram illustrating one embodiment of an MPS connection scheme. The MPS connection scheme 20 includes multiple DMRs and PEs and can be described as follows: Figure 30 In the MPS connection scheme 20, each PE is connected to four neighbor DMRs, and each DMR is connected to four neighbor PEs and four neighbor DMRs. Thus, the MPS connection scheme 20 may illustrate the PlanarA connection scheme discussed above.

[0385] To support high-bandwidth ports in the MPS connection scheme 20, the connections between ports (PE to DMR or DMR to DMR) can be short (i.e., limited to neighbors) and word-wide, meaning the number of electrical conductors (wires) in the data portion of the connection can be the same as the number of bits used in the ALU operands. PE to DMR connections can include address lines. DMR to DMR connections may not necessarily have address lines, but may have lines for flow control.

[0386] By keeping the PE nodes simple, a large array can be placed on a single VLSI IC at a modest cost (e.g., 16 rows by 16 columns = 256 PEs in one MPS embodiment). Suitable VLSI technologies may include, but are not limited to, complementary metal oxide semiconductor (CMOS) field effect transistors, with or without bipolar transistors in silicon or other semiconductors.

[0387] In some MPS implementations, communication between nodes can be under programmer control. In MPS, each PE can transfer data / instructions to adjacent DMRs and, optionally, communicate with other DMRs and PEs via those DMRs. This works well for transferring small amounts of data over short distances. However, for larger blocks of data or longer distances, it's more efficient to use a DMA engine to move data, freeing up the PEs to perform ALU operations.

[0388] For longer-distance block moves, some MPS embodiments can provide components for memory-to-memory transfers between DMRs without involving a PE. A PE can indirectly access a DMR-type port in a neighboring DMR through a special SM address associated with such a port. This allows the PE to create a new path for sending a message and subsequently tear down such a path, or alternatively receive a message. The PE can also store the data block to be transferred in an SM buffer in a neighboring DMR and then instruct the neighboring DMR to start a DMA operation through a special SM address associated with such an operation. This allows the PE to continue other tasks while the neighboring DMR coordinates the DMA transfer of data.

[0389] Various embodiments of the MPS can provide a favorable environment for executing useful algorithms. The algorithm of interest (e.g., for analyzing image data) can be decomposed into a flow chart of an ALU. Each flow chart can be mapped to the MPS array as a tree, a lattice, or any arbitrary network (including multiple feedback / feedforward paths). By combining several PEs and DMRs, the limited precision of an ALU can be extended to obtain accurate results for multiple words. When mapping a flow chart to the MPS, the communication delay between PE / DMR nodes, which is proportional to the distance between the nodes, may increase. In addition, if the communication queue is large or if reconfiguration is frequent, then the mapping may require more memory at each node. These factors can be compensated by careful programming that can take into account communication delays, queuing, and reconfiguration.

[0390] Systolic algorithms represent a class of algorithms that can be mapped particularly efficiently to various embodiments of MPS. Systolic algorithms have been developed for various applications in matrix arithmetic, image processing, and signal processing. In systolic algorithms, many processors can collaborate in a synchronized manner to perform difficult calculations. In an ideal algorithm implementation, each processor can repeatedly perform the same operation (or small operation loop) as long as the algorithm is needed, and data can flow through the network of processors through adjacent connections, balancing the production and consumption of data words. If each intermediate result data word produced is then immediately consumed by subsequent calculations, the amount of memory required can be minimized. Advantages of systolic algorithms can include the use of simplified processors, minimized memory requirements, and the ability to achieve high arithmetic operation rates using standard low-cost VLSI technology.

[0391] MPS embodiments can have many processors per chip and an entire MIMD architecture that can be configured to emulate the operation of other types of systems, such as SIMD systems and distributed MIMD systems. In some embodiments, MPS can run different algorithms in different areas of the chip simultaneously. Moreover, to save power, in some embodiments, programmers can selectively enable and disable clocks for at least some PEs and DMRs. Thus, unused PEs and DMRs can be disabled.

[0392] PE / DMR architecture

[0393] Figure 32 is a more detailed diagram illustrating one embodiment of the MPS architecture. Figure 32 In the FPGA, each PE is surrounded by four DMRs, with which it can communicate memory requests and messages. Each DMR is surrounded by four other DMRs, except that it is located near the fabric edge, where it may be adjacent to a chip I / O port. Each DMR can communicate with adjacent DMRs or chip I / O ports to establish a communication path and send / receive messages on that path.

[0394] MPS Operation

[0395] Figure 33 The diagram shows Figure 32 Example MPSs for an architecture example, consisting of an 8x8 array of PEs (squares) evenly interspersed with a 9x9 array of DMRs (circles). Programs can be compiled into tasks assigned to PEs. The first example program was compiled with taskID=62 and assigned to a specific PE in the upper left corner of the array. Variables (u, v, w) are declared as communication variables in the program source code and assigned to specific memory addresses in adjacent DMRs; u and v are buffers for I / O ports, and w is a buffer for on-chip network communication with its associated DMR. The second example program was compiled with taskID=71 and assigned to a specific PE within the array. Variable x is a declared communication variable and assigned to the DMR shown. The communication path associated with variable x runs from its assigned DMR through other DMRs to the I / O port in the top row. As shown, the two example programs do not communicate with each other, but they can be easily made to communicate by adding another communication variable to task 71 and a path between variable w in its DMR and the DMR adjacent to task 62.

[0396] ***

[0397] The embodiments of the present disclosure may be implemented in any of a variety of forms. For example, in some embodiments, the present invention may be implemented as a computer-implemented method, a computer-readable storage medium, or a computer system. In other embodiments, one or more custom-designed hardware devices (such as ASICs) may be used to implement the present invention. In other embodiments, one or more programmable hardware elements (such as FPGAs) may be used to implement the present invention.

[0398] In some embodiments, a non-transitory computer-readable memory medium may be configured such that it stores program instructions and / or data, wherein if the program instructions are executed by a computer system, the program instructions cause the computer system to perform a method, such as any one of the method embodiments described herein, or any combination of the method embodiments described herein, or any subset of any one of the method embodiments described herein, or any combination of such subsets.

[0399] In some embodiments, a computing device may be configured to include a processor (or a set of processors) and a memory medium, wherein the memory medium stores program instructions, wherein the processor is configured to read and execute the program instructions from the memory medium, wherein the program instructions are executable to implement any of the various method embodiments described herein (or any combination of the method embodiments described herein, or any subset of any of the method embodiments described herein, or any combination of such subsets). The device may be implemented in any of a variety of forms.

[0400] Although specific embodiments have been described above, even where only a single embodiment has been described with respect to a particular feature, these embodiments are not intended to limit the scope of the present disclosure. Unless otherwise stated, the examples of features provided in the present disclosure are intended to be illustrative rather than restrictive. The above description is intended to encompass such alternatives, modifications, and equivalents as will be apparent to those skilled in the art having the benefit of this disclosure.

[0401] The scope of the present disclosure includes any feature or combination of features disclosed herein (explicitly or implicitly), or any generalization thereof, whether or not it mitigates any or all of the problems addressed herein. Accordingly, new claims may be formulated during the prosecution of the present application (or an application claiming priority thereto) to any such combination of features. In particular, features in dependent claims may be combined with features in the independent claims with reference to the appended claims, and features in the respective independent claims may be combined in any appropriate manner and not just in the specific combinations listed in the appended claims.

Claims

1. A method for rendering an output image, the method comprising: receiving a plurality of input images, wherein each input image of the plurality of input images is taken from a different first position; receiving a view specification for rendering an output image, wherein the view specification includes a second position; rendering the output image based at least in part on the plurality of input images and the view specification, wherein the output image comprises an image of the area as seen from a second position, wherein the second position is different from each of the first positions, wherein rendering the output image based at least in part on the plurality of input images and the view specification comprises: For each pixel block in the plurality of pixel blocks of the output image: determining a mapping between a plurality of pixels along a perimeter of a respective pixel block and a plurality of corresponding pixels of each of one or more of the input images; and For each of the one or more input images: performing projective mapping of pixels located within a perimeter of corresponding pixels of the input image to corresponding pixels in the output image, wherein the projective mapping comprises linear extrapolation of a mapping between the plurality of pixels along a perimeter of a corresponding pixel block and the plurality of corresponding pixels of the input image, wherein a size of each pixel block of the plurality of pixel blocks is selected to balance a degree of image distortion introduced by the projective mapping and computational resources associated with rendering the output image to render the output image within a first latency, and wherein a smaller pixel block size reduces the degree of image distortion while utilizing a greater number of parallel computational resources to render the output image within the first latency relative to a larger pixel block size; and The output image is displayed on a display.

2. The method according to claim 1, further comprising: A series of output images is repeatedly rendered based on a series of corresponding multiple input images, wherein the series of output images are displayed on a display in real time to produce real-time video.

3. The method of claim 1, further comprising: Determine the projection surface; Wherein rendering the output image based at least in part on the plurality of input images comprises: mapping pixels of the plurality of input images onto the projection surface; as well as Positions on the projection surface are mapped to pixels of the output image.

4. The method of claim 3, further comprising: determining ranging information associated with an area surrounding the first location, Wherein said determining the projection surface is performed at least in part based on ranging information.

5. The method of claim 3, wherein the first location is a location on a vehicle, and The projection surface includes a horizontal portion surrounding the vehicle and a vertical portion surrounding the horizontal portion.

6. The method of claim 5, further comprising: determining a size of the horizontal portion based at least in part on a current speed of the vehicle, A higher current vehicle speed determines a larger horizontal portion than a lower current vehicle speed.

7. The method of claim 5, wherein the horizontal portion of the projection surface comprises an elliptical disk surrounding the vehicle, wherein the vertical portion of the projection surface comprises an elliptical cylinder surrounding an elliptical disk, and The projection surface further includes a middle area, which smoothly connects the elliptical disk and the elliptical cylinder.

8. The method of claim 7, wherein the major axis of the elliptical disk extends from the front of the vehicle to the rear of the vehicle, and The minor axis of the elliptical disk extends from the left side of the vehicle to the right side of the vehicle.

9. The method of claim 3, further comprising: determining the geometric layout of the environment surrounding the first location, Wherein the projection surface comprises a basic spline surface approximating an environment surrounding the first location.

10. The method of claim 1, wherein the plurality of pixels along a perimeter of a corresponding pixel block comprises four corner pixels of the corresponding pixel block.

11. The method of claim 1 , further comprising: receiving user input specifying one or more second view specifications; rendering one or more second output images based at least in part on the one or more second view specifications and the plurality of input images; as well as The one or more second output images are displayed on the display.

12. The method of claim 1 , wherein at least a portion of the output image is derived from two or more of the plurality of input images; and Wherein rendering at least a portion of the output image comprises blending contributions of the two or more input images from the plurality of input images.

13. The method of claim 12, wherein the blending is weighted based at least in part on a proximity of a first position of the two or more input images to a projected position of the at least one portion of the output image.

14. The method of claim 12, wherein the blending corrects for lighting differences between the two or more input images.

15. The method of claim 1 , wherein the view specification further comprises one or more of: an azimuth measured from the origin at the second position; a polar angle measured from the origin at the second position; Field of view; Projected surface depth; one or more projection surface parameters; Pixel resolution; as well as Mixed width.

16. A non-transitory computer-readable memory medium comprising program instructions that, when executed by one or more processors, cause a computing system to: receiving a plurality of input images, wherein each input image of the plurality of input images is taken from a different first position; receiving a view specification for rendering an output image, wherein the view specification includes a second position; Determine the projection surface; rendering an output image based at least in part on the plurality of input images, the view specification, and the projection surface, wherein the output image comprises an image of the area as seen from a second position, wherein the second position is different from each of the first positions, wherein rendering the output image based at least in part on the plurality of input images and the view specification comprises: For each pixel block in the plurality of pixel blocks of the output image: determining a mapping between a plurality of pixels along a perimeter of a respective pixel block and a plurality of corresponding pixels of each of one or more of the input images; For each of the one or more input images: performing projective mapping of pixels located within a perimeter of corresponding pixels of the input image to corresponding pixels in the output image, wherein the projective mapping comprises a linear extrapolation of a mapping between the plurality of pixels along a perimeter of a corresponding pixel block and the plurality of corresponding pixels of the input image, wherein a size of each of the plurality of pixel blocks is selected to balance a degree of image distortion introduced by the projective mapping and computational resources associated with rendering the output image to render the output image within a first latency, and wherein a smaller pixel block size reduces the degree of image distortion while utilizing a greater number of parallel computational resources to render the output image within the first latency relative to a larger pixel block size.

17. The non-transitory computer-readable memory medium of claim 16, wherein the program instructions are further executable by the one or more processors to cause the computing system to: determining the geometric layout of the environment surrounding the first location, Wherein the projection surface comprises a basic spline surface approximating an environment surrounding the first location.

18. A computing system comprising: one or more memories; as well as one or more processors coupled to the one or more memories, wherein the one or more memories store program instructions that, when executed by the one or more processors, cause the computing system to: receiving a plurality of input images, wherein each input image of the plurality of input images is taken from a different first position; receiving a view specification for rendering an output image, wherein the view specification includes a second position; rendering an output image based at least in part on the plurality of input images and the view specification, wherein the output image comprises an image of the area as seen from a second position, wherein the second position is different from each of the first positions, wherein to render the output image, the computing system is configured to: for each pixel block of a plurality of pixel blocks of the output image: determining a mapping between a plurality of pixels along a perimeter of a respective pixel block and a plurality of corresponding pixels of each of one or more of the input images; For each of the one or more input images: Performing projective mapping of pixels located within a perimeter of corresponding pixels of the input image to corresponding pixels in the output image, wherein the projective mapping comprises a linear extrapolation of a mapping between the plurality of pixels along a perimeter of the corresponding pixel block and the plurality of corresponding pixels of the one or more input images, wherein a size of each of the plurality of pixel blocks is selected to balance a degree of image distortion introduced by the projective mapping and computational resources associated with rendering the output image to render the output image within a first latency, and wherein a smaller pixel block size reduces the degree of image distortion while utilizing a greater number of parallel computational resources to render the output image within the first latency relative to a larger pixel block size.

Images