Improved fractional interpolation for convolution processors in autonomous or semi-autonomous systems - Patents.com

Abstract

A system and method for improved fractional interpolation for a convolution processor in an autonomous or semi-autonomous system. An exemplary method includes acquiring images from an image sensor disposed around a vehicle. For a first image, an interpolation sequence is determined for interpolating the first image according to fractional interpolants, the interpolation sequence including a combination of respective integer interpolations and integer downsampling. The first image is interpolated according to the interpolation sequence.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 316,847, filed March 4, 2022, entitled “ENHANCED FRACTIONAL INTERPOLATION FOR CONVOLUTIONAL PROCESSOR IN AUTONOMOUS OR SEMI-AUTONOMOUS SYSTEMS,” the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to hardware processors, and more particularly, to the use of hardware processors for fractional interpolation. [Background technology]

[0003] Interpolation is commonly relied upon to estimate new data points based on known data points. For example, interpolation can be used to adjust the resolution of the images forming the video presented on a television. In this example, the television may be at a first resolution (e.g., 3,840 pixels by 2,160 pixels) and the video source may be at a second resolution (e.g., 1,920 pixels by 1,080 pixels). During presentation of the video source, the television can interpolate the video source to output at the first resolution. As can be appreciated, this interpolation is an integer multiple of the second resolution.

[0004] Various techniques may be utilized to perform integer interpolation of data (e.g., an image). For example, an image may be upsampled by replicating pixels of the image a number of times based on a particular integer. In this example, each pixel may be doubled, quadrupled, etc. Additionally, integer interpolation may be performed using bilinear interpolation. As known to those skilled in the art, bilinear interpolation is performed using linear interpolation in a first direction and then again in a second direction. [Brief description of the drawings]

[0005] [Figure 1A] FIG. 1 is a block diagram illustrating an exemplary vehicle processor system for a vehicle that outputs an image to which fractional interpolation has been applied.

[0006] [Figure 1B] FIG. 2 is a block diagram of an input image being interpolated and downsampled according to an interpolation sequence.

[0007] [Figure 2A] 1 is a flowchart of an exemplary process for performing fractional interpolation using a vehicle processor system.

[0008] [Figure 2B] FIG. 2 is a block diagram illustrating a convolution engine of an exemplary vehicle processor system that performs an interpolation sequence according to fractional interpolants.

[0009] [Diagram 3] 1 is a flowchart of an exemplary process for interpolating images received from an image sensor included in an autonomous or semi-autonomous vehicle.

[0010] [Figure 4] 1 is a block diagram illustrating an example vehicle including a vehicle processor system.

[0011] Embodiments of the present disclosure and their advantages are best understood by reference to the following detailed description: In one or more of the figures, like reference numerals have been used to identify like elements illustrated, it being understood that such illustrations are for the purpose of illustrating embodiments of the present disclosure and are not intended to limit the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] This application describes improved techniques for performing fractional interpolation of input data, such as images, video sequences, and the like. As may be understood with respect to images, fractional interpolation allows fine tuning of image resolution by allowing increments expressed as fractional increments (e.g., 4 / 3, 5 / 8, 63 / 33, etc.). Details are described herein for performing fractional interpolation using efficient computational techniques. These techniques may reduce the time required to perform fractional interpolation. Additionally, these techniques may allow for a reduction in hardware components required to efficiently perform fractional interpolation. In some embodiments, these techniques may be particularly suited for improved fractional interpolation using convolution engines or convolution processing hardware, where the interpolation is performed based on convolution techniques.

[0013] As described below, fractional interpolation can enable an autonomous or semi-autonomous vehicle to quickly adjust the resolution of images received from an image sensor located on the vehicle. Fractional interpolation can enable complex deep learning models to receive inputs (e.g., images) of a particular size, regardless of the size of the image sensor located on the vehicle. The particular size can include all images being of the same (e.g., fixed) resolution, or some images being of a higher resolution compared to other images. For example, images from a forward-looking image sensor may have a higher resolution compared to images from an image sensor located on the side of the vehicle. Thus, forward-looking images may be downsampled and / or side-looking images may be upsampled.

[0014] The computationally efficient fractional interpolation techniques described herein allow for rapid adjustment of hardware elements included in a vehicle. For example, newer image sensors may be installed in a newer version of a vehicle. In this example, fractional interpolation may be used to adjust the resolution of the output from these newer image sensors. As another example, newer image sensors may be installed in an existing vehicle in combination with older image sensors. In this example, fractional interpolation may be used to adjust the output from these image sensors to the same resolution.

[0015] Additionally, the techniques described herein can enable rapid implementation of new versions of machine learning models in vehicles through efficient adjustment of the resolution of the input images. For example, newer versions of machine learning models can use input images with higher resolution. In this example, fractional interpolation can be applied to adjust the resolution of the input images.

[0016] Interpolation can adjust the number of elements included in the input data (e.g., an input image). With respect to the example of an input image, interpolation can allow the number of pixels included in the input image to be increased. Thus, the resolution of the input image can be adjusted upward using interpolation. As can be appreciated, this upward adjustment of the resolution estimates new data to include in the output. Although images are described herein, as can be appreciated, the input data can be output from a previous layer of the neural network. For example, the input data can be a data blob, tensor, etc., provided to a layer of a neural network (e.g., a convolutional neural network) to compute a forward pass through the layer.

[0017] Typically, the interpolation is performed using integer multiples ("integer interpolation"). For example, integer interpolation may allow for doubling, tripling, quadrupling, etc. of the resolution of the input image. To perform such integer interpolation, various techniques may be utilized, as known to those skilled in the art, such as nearest neighbor, bilinear, bicubic, spline, etc. Furthermore, the input image may be interpolated using a convolution in which a window (e.g., a filter or kernel) allows for the estimation of new pixels to be included in the input image. For example, a bilinear interpolation may be performed using a convolution in which a kernel is centered on a pixel and weights the neighboring pixels. As can be appreciated, such a convolution technique may be referred to as, by way of example, a transposed convolution or a deconvolution.

[0018] Similarly, downsampling may use techniques such as averaging pixel values ​​within a window (e.g., 2x2 pixels). Thus, in some embodiments, downsampling may utilize convolution techniques (e.g., pooling, averaging values ​​within a window, etc.). Additional exemplary downsampling techniques include box sampling, mipmap, sinc, using strides, etc.

[0019] Advantageously, such convolution-based integer interpolation, and optionally downsampling, may be efficiently implemented in a processor used to compute a forward pass through a machine learning model (e.g., a neural network model). Exemplary processors that may be used to perform at least efficient convolution of input data with a window include the processors (e.g., matrix processors) described in U.S. Pat. No. 11,157,287, U.S. Patent Application Publication No. 2019 / 0026250, and U.S. Pat. No. 11,157,441, the entireties of which are incorporated herein by reference and form part of this disclosure as if set forth herein.

[0020] However, such integer interpolation techniques may be too flexible for the needs of more generalized interpolation. As described above, an autonomous or semi-autonomous vehicle may, at least in part, use image sensors (e.g., cameras) to obtain input data used by machine learning models in performing autonomous or semi-autonomous driving. These image sensors may be positioned around the vehicle, for example, as shown in FIG. 1A, and thus obtain images used by the machine learning models. The image sensors may output images of various resolutions, for example, due to availability of image sensors or due to the level of detail required for the individual image sensors. Thus, it may be advantageous to adjust the size (e.g., resolution) of the images from the image sensors. As can be appreciated, such adjustments may not be strictly integer-based. For example, a first image sensor may be 4 / 3 the size of a second image sensor.

[0021] As mentioned above, fractional interpolation can allow for the adjustment of fractional ratios. However, implementing such fractional adjustments may typically require complex algorithms or dedicated processor hardware elements. For convolution engines or processors, such fractional ratios cause technical problems because the number of pixels in each window is not fixed and the weights depend on the pixel's position in the image. Therefore, complex hardware control may be required to ensure that fractional interpolation is implemented quickly.

[0022] In contrast, the techniques described herein allow for the determination of an interpolation sequence that can be implemented efficiently. In embodiments in which a convolution engine or processor is used, the interpolation sequence can allow for computationally efficient execution of fractional interpolation. As discussed above, the interpolation sequence includes one or more combinations of interpolation (e.g., upsampling) and downsampling. For example, the interpolation sequence can include two combinations. In this example, the first combination can cause an interpolation at an integer multiple followed by downsampling at an integer fraction. As an example, the interpolation can increase the resolution of the image by a factor of three, and then the downsampling reduces the resolution of the image by a factor of two. In this way, the input image is effectively fractionally interpolated by a factor of 1.5 (e.g., 3 / 2). The second combination can then follow the first combination. For example, the second combination can interpolate by a factor of four and then downsample by a factor of three). In this example, the output from the first combination is fractionally interpolated by a factor of 5 / 3. Thus, the input image to the interpolation sequence is fractionally interpolated by a factor of 15 / 6.

[0023] The above-mentioned fractional interpolation allows efficient use of processors with successive interpolation and downsampling rounds. The techniques described herein can provide enormous technological savings in terms of graphical processing units (GPUs), neural processing units (NPUs), processors implementing convolution engines, hardware application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs), etc.

[0024] Block Diagram FIG. 1A is a block diagram illustrating an example vehicle processor system 100 of a vehicle 102 that outputs an image (e.g., image N) to which fractional interpolation has been applied. As shown, the vehicle includes multiple image sensors (e.g., cameras 104A-104N) disposed around the vehicle 102. The cameras 104A-104N can provide a substantially 360 degree field of view around the vehicle 102 such that autonomous or semi-autonomous driving can be performed. The vehicle 102 may represent a consumer vehicle, such as an electric vehicle, or a commercial vehicle (e.g., truck, van, etc.). The vehicle 102 may further represent an unmanned vehicle, such as an unmanned aerial vehicle, an unmanned ground vehicle, etc.). In some embodiments, the functionality described herein may be implemented by a robot that includes one or more cameras.

[0025] The vehicle includes a vehicle processor system 100 of one or more processors that implement machine learning models to autonomously or semi-autonomously drive the vehicle 102. For example, the machine learning models may include convolutional neural networks, fully connected neural networks, etc. As part of the process of implementing autonomous or semi-autonomous driving, the vehicle processor system 100 acquires images from cameras 104A-104N.

[0026] As can be appreciated, the captured images may need to be adjusted in size (e.g., resolution) to be used by the vehicle processor system 100 to perform autonomous or semi-autonomous driving. For example, the cameras 104A-104N output images at various resolutions. As an example, the camera 104A may output images at a first resolution, and the camera 104N may output images at a second, lower resolution. As another example, the machine learning model implemented by the vehicle processor system 100 may use input information at a particular resolution. For example, the machine learning model may use vectorized images as input to an initial layer or portion of the model, or to an intermediate layer or subsequent portion of the model. In this example, the size of the vectors may be fixed. Thus, the images may need to be fractionally interpolated to arrive at the size of the vectors.

[0027] Exemplary images are shown as being provided from an image sensor (e.g., via a bus or other communication technology) to vehicle processor system 100. Specifically, image 106A from camera 104A and image 106N from camera 104N are shown as being received by vehicle processor system 100. In the illustrated embodiment, vehicle processor system 100 fractionally interpolates image 106A to correspond to the resolution of image 106A. Thus, images 106A and 106N can be used as equal-sized inputs (e.g., equal number of elements or pixels) to a machine learning model.

[0028] As described in more detail below, the vehicle processor system 100 can fractionally interpolate the image 106N at a particular fractional value using an interpolation sequence. For example, the interpolation sequence can include one or more combinations of respective interpolation and downsampling. In this example, the number of combinations can be based on the fractional value applied to the image 106N. For example, if the fractional value can be achieved by one combination, the vehicle processor system 100 can perform one combination of interpolation and downsampling. An exemplary fractional value can include 4 / 3, such that the vehicle processor system 100 interpolates the image N 106N by a factor of four (e.g., upsamples by a factor of four) and downsamples by a factor of three.

[0029] In some embodiments, the vehicle processor system 100 can limit the range to which interpolation and downsampling can be applied in a combination to the image 106N. For example, the vehicle processor system 100 may apply interpolation up to a first threshold integer (e.g., 4, 8, 12, 32, 64, 128) and apply downsampling up to a second threshold integer (e.g., 4, 8, 12, 32, 48, 64, etc.). Thus, for some fractional values, the vehicle processor system 100 can apply two or a combination of interpolation and downsampling.

[0030] As an example of two or more combinations, the fractional value may be 35 / 16. For this fractional value, the vehicle processor system 100 may determine to use two combinations, the first representing causing an interpolation of the value 7 / 4 times and the second representing causing an interpolation of the value 5 / 4 times. This determination may be made based on the number "35" being greater than a threshold value that the vehicle processor system 100 is configured to interpolate.

[0031] For some fractional values, the vehicle processor system 100 can determine the use of a combination of threshold numbers to arrive at a value substantially close to the fractional value. In some embodiments, the threshold number may be a combination of two. As can be seen, by using two combinations, a significant number of fractional values ​​can be achieved. In some embodiments, the vehicle processor system 100 applies interpolation up to a first threshold number and downsamples up to a second threshold number. Thus, there may be fractional values ​​that cannot be accurately achieved with two combinations. In these embodiments, the vehicle processor system 100 can determine the value that is closest to the desired fractional value. In some embodiments, the threshold number combination may be 3, 5, 7, 9. In some embodiments, there may be no threshold number combination.

[0032] To ensure accuracy of the interpolation sequence, the vehicle processor system 100 may begin with an interpolation to increase the resolution of the image 106N. This interpolation may then be followed by downsampling. In some embodiments, the interpolation sequence may initially include two or more interpolations. In some embodiments, the interpolation may be followed by two or more downsamplings.

[0033] While Figure 1A illustrates interpolating an image 106 to increase its resolution, it can be appreciated that the techniques described herein can allow for a reduction in resolution. For example, Figure 1B illustrates an example of fractional interpolation to reduce the resolution of an image.

[0034] 1B is a block diagram of an input image 156 being interpolated and downsampled according to an interpolation sequence. As shown, the interpolation sequence fractionally interpolates the input image 152 by a factor of 2 / 3. Thus, in this example, the resolution of the input image 152 is reduced.

[0035] The vehicle processor system 100 can determine an interpolation sequence based on the fractional value 2 / 3, which can be achieved with one combination of interpolation and downsampling, so the system 100 specifies that the image 152 is to be interpolated by a factor of 2, and then downsampled by a factor of 3.

[0036] In some embodiments, the vehicle processor system 100 can perform integer interpolation using a convolution technique. In this example, a window can be moved across multiple pixels of the image 152, and weights can be applied to pixels near the center pixel of the window. These weights can then be used to estimate new pixels to include in the image 152. In some embodiments, the image 152 may be interpolated according to other techniques. A convolution engine or processor can be used to perform this integer interpolation as described herein. As mentioned above, the engine or processor can include a matrix processor (e.g., a non-systolic array) that calculates the product of the weights and the input data (e.g., the weights may be placed on one axis of a matrix and the input data on the other axis). In some embodiments, the maximum interpolation value can be based on the size of the matrix.

[0037] A temporary image 154 is shown as a result of the 2x interpolation. This image may be stored, for example, by a convolution engine or processor (e.g., cache, temporary storage, etc.). The vehicle processor 100 may then downsample the temporary image 154 by a factor of 3. For example, the temporary image 154 may be used as input data to a convolution engine or processor. As above, in some embodiments, the downsampling may be performed using convolution techniques. For example, a window may be moved across multiple pixels of the image 154 and multiple pixels within the window may be averaged or otherwise combined to reduce the number of pixels. The resulting output 156 is shown in FIG. 1B and represents a 2 / 3x fractional interpolation.

[0038] flowchart 2A is a flow chart of an example process 200 for performing fractional interpolation using a vehicle processor system. For convenience, the process 200 is described as being performed by a system of one or more processors (e.g., vehicle processor system 100). In some embodiments, the system may be included in a vehicle (e.g., vehicle 102, 400). In some embodiments, the system may represent a portion of a processor (e.g., a convolution engine or convolution processor).

[0039] At block 202, the system acquires images for interpolation. As described above, the system may be in communication (e.g., wired or wireless communication) with a camera used to acquire images for processing by the system. For example, the system may be used to implement autonomous or semi-autonomous operation of a vehicle.

[0040] At block 204, the system specifies a fractional value associated with the interpolation of the image. The system can obtain information indicative of the fractional value to be applied. For example, the system can implement a pipeline for pre-processing images received from a camera. In this example, the system can access information indicative of the fractional value. For example, the system can execute code that causes the interpolation of the image according to the fractional value.

[0041] At block 206, the system determines an interpolation sequence based on the fractional values. As described above, the interpolation sequence can include one or more combinations of interpolation and downsampling. Thus, the interpolation sequence can include a first combination including a first interpolation followed by a second downsampling. The interpolation sequence can also include a second combination including a second interpolation followed by a second downsampling. In some embodiments, the system can limit the number of combinations to two.

[0042] To determine the interpolation sequence, the system may have access to information indicating respective integer thresholds for interpolation and downsampling. Thus, the system may limit integer interpolation to a maximum of 8, 16, 32, 64, 80 times. The system may also limit downsampling to a maximum of 8, 16, 32, 64, 80 times. These limitations may be based on hardware constraints of the system, such as hardware constraints of a convolution engine or processor. The limitations may also be defined in software or code that the system is running.

[0043] Thus, for an integer threshold, the system can determine whether a single combination can result in a fractional value. For example, if the fractional value is 8 / 5, the system can include a single combination in the interpolation sequence. This single combination includes interpolating the image by a factor of 8 and then downsampling the image by a factor of 5.

[0044] For fractional values ​​that require two or more combinations, the system can determine respective interpolation and downsampling values ​​for these combinations. For example, for two combinations, the system can separate the fractional value into two fractions that, when multiplied, equal the fractional value. In some embodiments, the system can determine numerator and denominator multiples. These multiples can then be used to select respective interpolation and downsampling values. As mentioned above, some fractional values ​​may not be achieved exactly, and the system can select values ​​that cause substantially the closest interpolation to be applied.

[0045] The system applies the determined interpolation sequence at block 208. As described above, the system uses the interpolation sequence to interpolate according to the fractional values.

[0046] FIG. 2B is a block diagram illustrating a convolution engine 250 of the example vehicle processor system 100 that performs an interpolation sequence 252 according to a fractional interpolant 254.

[0047] In the illustrated embodiment, the convolution engine 250 receives the image 256 and interpolates the image according to the fractional interpolants 254. The vehicle processor system 100 or the convolution engine 250 may determine an interpolation sequence 252 that causes the interpolation with the fractional interpolants 254. As shown in FIG. 2B, the interpolation sequence 252 includes two combinations of interpolation and downsampling.

[0048] As described above, the convolution engine 250 can perform interpolation using convolution techniques. In addition, the convolution engine 250 can perform downsampling using convolution techniques. Thus, the image 256 can be interpolated by the engine 250, and the output is returned to the convolution engine 250 for subsequent downsampling by the interpolation sequence 252.

[0049] 3 is a flow chart of an example process 300 for interpolating images received from an image sensor included in an autonomous or semi-autonomous vehicle. For convenience, the process 300 is described as being performed by a system of one or more processors (e.g., vehicle processor system 100). In some embodiments, the system may be included in a vehicle (e.g., vehicle 102, 400). In some embodiments, the system may represent a portion of a processor (e.g., a convolution engine or convolution processor).

[0050] In block 302, the system acquires an input image from an image sensor. As shown in Figure 1A, the system may acquire images from cameras positioned around the vehicle.

[0051] At block 304, the system performs fractional interpolation on at least a portion of the input images. The system determines whether images from each camera should be fractionally interpolated. For example, the cameras can operate at various resolutions such that the resolutions of the images may vary. In some embodiments, the system accesses the specifications of each camera to identify the resolutions. In these embodiments, the system can cause an interpolation of the resolutions of the input images so that they match (e.g., the system can use the highest resolution, the lowest resolution, the average resolution, etc.). In some embodiments, the system can implement a pre-processing pipeline that indicates that fractional interpolation should be performed on images from a subset of the image sensors. In this case, the system applies a particular fractional interpolation value to at least a portion of the input images.

[0052] At block 306, the system pre-processes the input image. The system can optionally apply additional pre-processing such as filtering, brightness adjustment, etc.

[0053] At block 308, the system provides the input images to a machine learning model or deep learning pipeline. The images, including at least some of the fractionally interpolated images, are then provided to the machine learning model. In some embodiments, the machine learning model can be used to perform autonomous or semi-autonomous driving.

[0054] Vehicle block diagram 4 illustrates a block diagram of a vehicle 400 (e.g., vehicle 102). The vehicle 400 may include one or more electric motors 402 that move the vehicle 400. The electric motors 402 may include, for example, induction motors, permanent magnet motors, etc. As known to those skilled in the art, batteries 404 (e.g., one or more battery packs each including multiple batteries) may be used to power the electric motors 402.

[0055] The vehicle 400 further includes a propulsion system 406 that can be used to set the gear (e.g., propulsion direction) of the vehicle. For an electric vehicle, the propulsion system 406 can regulate the operation of the electric motor 402 to change the propulsion direction.

[0056] Additionally, the vehicle includes a vehicle processor system 100 configured to fractionally interpolate data, such as images, received from an image sensor (e.g., cameras 104A-104N) of the vehicle 400. The vehicle processor system 100 can further output information to a display 408 included in the vehicle 400 and receive information (e.g., user input) from the display 408.

[0057] Other embodiments All of the processes described herein may be embodied and fully automated via software code modules executed by a computing system including one or more computers or processors. The code modules may be stored in any type of non-transitory computer readable medium or other computer storage device. Parts or all of the methods may be implemented in dedicated computer hardware.

[0058] Many other variations beyond those described herein will be apparent from this disclosure. For example, depending on the embodiment, certain operations, events, or functions of any of the algorithms described herein may be performed in a different order, added, combined, or omitted entirely (e.g., not all operations or events described may be necessary to implement an algorithm). Furthermore, in certain embodiments, operations or events may be performed simultaneously rather than sequentially, for example, via multi-threading, interrupt processing, or multiple processors or processor cores, or on other parallel architectures. Furthermore, different tasks or processes may be performed by different machines and / or computing systems that can function together.

[0059] The various example logic blocks, modules, and engines described in connection with the embodiments disclosed herein may be implemented or executed by a machine, such as a processing unit or processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processor may be a microprocessor, but in alternative examples, the processor may be a controller, a microcontroller, or a state machine, combinations thereof, and the like. The processor may include electrical circuitry configured to process computer-executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, the processor may also include primarily analog components. For example, some or all of the signal processing algorithms described herein may be implemented with analog circuitry or mixed analog and digital circuitry. The computing environment may include any type of computer system, including, but not limited to, a computer system based on a computational engine within a microprocessor, mainframe computer, digital signal processor, portable computing device, device controller, or appliance, to name a few.

[0060] In particular, conditional language such as "can," "could," "might," or "may" is understood within the context in which it is generally used to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless otherwise specified. Thus, such conditional language does not generally imply that the features, elements, and / or steps are in any way required by one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps should be included in or performed in any particular embodiment, with or without user input or prompting.

[0061] Disjunctive language, such as the phrase "at least one of X, Y, and Z," is understood in the context of common usage to indicate that an item, term, etc. can be either X, Y, or Z, or a combination thereof (e.g., X, Y, and / or Z), unless otherwise indicated. Thus, such disjunctive language is generally not intended to, and should not, imply that a particular embodiment requires at least one of X, at least one of Y, and at least one of Z, respectively, to be present.

[0062] Any process descriptions, elements or blocks in the flow diagrams described herein and / or illustrated in the accompanying drawings should be understood as potentially representing modules, segments or portions of code that include one or more executable instructions for implementing a particular logical function or element in the process. As will be appreciated by those skilled in the art, alternative implementations in which elements or functions may be omitted, performed or described in a different order than that shown or described, including substantially simultaneously or in reverse order, depending on the functionality involved, are included within the scope of the embodiments described herein.

[0063] Unless otherwise noted, articles such as "a" or "an" should generally be construed to include one or more of the described items. Thus, phrases such as "a device configured to" are intended to include one or more of the described devices. Such one or more described devices may also be collectively configured to perform the stated description. For example, "a processor configured to perform descriptions A, B, and C" may include a first processor configured to perform description A working in conjunction with a second processor configured to perform descriptions B and C.

[0064] It should be emphasized that many variations and modifications can be made to the above-described embodiments, and that these elements should be understood as being among other acceptable examples, and all such modifications and variations are intended to be included within the scope of the present disclosure.

Claims

1. 1. A computer-implemented method comprising: acquiring images from a plurality of image sensors positioned around the vehicle; determining an interpolation sequence for a first image to interpolate the first image according to fractional interpolants, the interpolation sequence including one or more combinations of respective integer interpolations and integer downsampling; Interpolating the first image according to the interpolation sequence; A method comprising:

2. 2. The method of claim 1 , wherein the interpolation sequence includes a first combination including a first interpolation followed by a first downsampling, and further includes a second combination receiving an output from the first combination, the second combination including a second interpolation followed by a second downsampling.

3. The method of claim 1 , wherein the interpolation is performed via a convolution engine or processor.

4. The method of claim 3 , wherein the convolution engine or the convolution processor comprises a matrix of computational elements, the matrix of computational elements being non-systolic.

5. The method of claim 1 , wherein interpolating the first image comprises interpolating the first image and downsampling the first image using a convolution process.

6. The method of claim 1 , wherein the integer interpolation value when multiplied is equal to the numerator of the fractional interpolation value.

7. The method of claim 1 , wherein the integer downsampling value when multiplied is equal to a denominator of the fractional interpolation value.

8. 1. A vehicle processor system configured for inclusion in a vehicle, comprising: Acquire images from a plurality of image sensors arranged around the vehicle; determining an interpolation sequence for the first image to interpolate the first image according to fractional interpolations, the interpolation sequence including one or more combinations of respective integer interpolations and integer downsampling; A vehicle processor system configured to interpolate the first image according to the interpolation sequence using a convolution engine of the vehicle processor system.

9. 9. The system of claim 8, wherein the interpolation sequence includes a first combination including a first interpolation followed by a first downsampling, and further includes a second combination receiving an output from the first combination, the second combination including a second interpolation followed by a second downsampling.

10. The system of claim 8 , wherein the interpolation is performed via a convolution engine or processor.

11. The system of claim 10 , wherein the convolution engine or the convolution processor comprises a matrix of computational elements, the matrix of computational elements being non-systolic.

12. 9. The system of claim 8, wherein interpolating the first image comprises interpolating the first image and downsampling the first image using a convolution process.

13. The system of claim 7 , wherein the integer interpolant value when multiplied is equal to a numerator of the fractional interpolant value.

14. The system of claim 8 , wherein the integer downsampling value when multiplied is equal to a denominator of the fractional interpolation value.

15. A non-transitory computer storage medium storing instructions, comprising: The instructions, when executed by a system of one or more processors, cause the one or more processors to: Acquiring images from a plurality of image sensors disposed around the vehicle; determining an interpolation sequence for the first image to interpolate the first image according to fractional interpolations, the interpolation sequence including one or more combinations of respective integer interpolations and integer downsampling; and a non-transitory computer storage medium for causing a convolution engine of the vehicle processor system to interpolate the first image according to the interpolation sequence.

16. 16. The computer storage medium of claim 15, wherein the interpolation sequence includes a first combination including a first interpolation followed by a first downsampling, and further includes a second combination receiving an output from the first combination, the second combination including a second interpolation followed by a second downsampling.

17. 16. The computer storage medium of claim 15, wherein the interpolation is performed via a convolution engine or processor.

18. 16. The computer storage medium of claim 15, wherein interpolating the first image comprises interpolating the first image and downsampling the first image using a convolution process.

19. 16. The computer storage medium of claim 15, wherein the integer interpolant value when multiplied is equal to the numerator of the fractional interpolant value.

20. 16. The computer storage medium of claim 15, wherein the integer downsampling value when multiplied is equal to a denominator of the fractional interpolation value.

Images