Enhanced score interpolation for convolution processors in autonomous or semi-autonomous systems
By using efficient computing technology and interpolation sequences of convolution engines in autonomous or semi-autonomous vehicles, fractional interpolation of image sensor data is realized, the problem of inconsistent image resolution is solved, computing performance and hardware efficiency are improved, and new versions are supported. Efficient input data processing of machine learning models.
Patent Information
- Application Number
- CN202380034346.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-04
- Filing Date
- 2023-03-02
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, in autonomous or semi-autonomous vehicles, the image resolution output by the image sensor is inconsistent and difficult to adjust through integer interpolation, resulting in the need of complex algorithms or specialized hardware, and the convolution engine or processor faces complex hardware control when fractional interpolation is performed. challenge.
Using efficient computing technology and convolution engine, fractional interpolation is achieved through interpolation sequences (including combinations of upsampling and downsampling), allowing fine-grained adjustment of image resolution, and is suitable for image sensor data processing in autonomous or semi-autonomous vehicles.
It realizes rapid adjustment of image resolution, reduces the demand for hardware components, improves computing performance, is suitable for image sensor data processing with different resolutions, and supports efficient input data processing of new versions of machine learning models.
Smart Images

Figure CN119072711A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Patent Provisional Application No. 63 / 316,847, filed on March 4, 2022, entitled “ENHANCED FRACTIONAL INTERPOLATION FOR CONVOLUTIONAL PROCESSOR IN AUTONOMOUS OR SEMI-AUTONOMOUS SYSTEMS,” the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present disclosure relates to a hardware processor, and more particularly to the use of a hardware processor for fractional interpolation. Background Art
[0004] Interpolation is often relied upon to estimate new data points based on known data points. For example, interpolation can be used to adjust the resolution of an image forming a video presented on a television. In this example, the television can be at a first resolution (e.g., 3840 pixels by 2160 pixels) while the video source is at a second resolution (e.g., 1920 pixels by 1080 pixels). During presentation of the video source, the television can interpolate the video source for output at the first resolution. As can be appreciated, the interpolation is an integer multiple of the second resolution.
[0005] To perform integer interpolation of data (e.g., an image), different techniques may be utilized. For example, an image may be upsampled by replicating the pixels of the image multiple times based on a particular integer. In this example, each pixel may be doubled, quadrupled, etc. Additionally, integer interpolation may be performed using bilinear interpolation. As known to those skilled in the art, bilinear interpolation is performed using linear interpolation in a first direction and then again using linear interpolation in a second direction. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1A is a block diagram illustrating an example vehicle processor system for a vehicle that outputs an image to which fractional interpolation is applied.
[0007] Figure 1B is a block diagram for interpolating and downsampling an input image according to an interpolation sequence.
[0008] Figure 2A is a flow chart of an example process for performing fractional interpolation using a vehicle processor system.
[0009] Figure 2B is a block diagram illustrating a convolution engine of an example vehicle processor system that performs an interpolation sequence based on fractional interpolation values.
[0010] Figure 3 is a flow chart of an example process for interpolating images received from an image sensor included on an autonomous or semi-autonomous vehicle.
[0011] Figure 4 is a block diagram illustrating an example vehicle including a vehicle processor system.
[0012] Embodiments of the present disclosure and their advantages can be best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more figures, wherein the illustrations in the figures are for illustrating embodiments of the present disclosure, rather than for limiting the present disclosure. DETAILED DESCRIPTION
[0013] The present application describes enhanced techniques for performing fractional interpolation on input data, such as images, video sequences, etc. Fractional interpolation as shown with respect to images allows fine-grained adjustment of image resolution by allowing increases represented by fractional increases (e.g., 4 / 3, 5 / 8, 63 / 33, etc.). Details for performing fractional interpolation using efficient computational techniques are described herein. These techniques can reduce the time associated with performing fractional interpolation. In addition, these techniques can reduce the hardware components required for efficiently performing fractional interpolation. In some embodiments, these techniques can be particularly suitable for enhanced fractional interpolation using a convolution engine or convolution processing hardware, where interpolation is performed based on convolution techniques.
[0014] As will be described, fractional interpolation can allow an autonomous or semi-autonomous vehicle to quickly adjust the resolution of images received from an image sensor located on the vehicle. Fractional interpolation can enable complex deep learning models to receive inputs (e.g., images) of a specific size, regardless of the size of the image sensor located on the vehicle. The specific size can include all images having the same (e.g., fixed) resolution or certain images having a higher resolution than other images. For example, an image from a front view image sensor can have a higher resolution than an image from an image sensor located on the side of the vehicle. Therefore, the front view image can be downsampled and / or the side view image can be upsampled.
[0015] By using the computationally efficient fractional interpolation techniques described herein, hardware elements included on a vehicle can be quickly adjusted. For example, newer image sensors can be installed in newer versions of vehicles. In this example, fractional interpolation can be used to adjust the resolution of the output of these newer image sensors. As another example, newer image sensors can be installed in conjunction with older image sensors on an existing vehicle. For this example, fractional interpolation can be used to adjust the output of these sensors to the same resolution.
[0016] In addition, the techniques described in this article can quickly implement new versions of machine learning models in vehicles by efficiently adjusting the resolution of input images. For example, a newer version of a machine learning model can use a higher resolution input image. In this example, fractional interpolation can be applied to adjust the resolution of the input image.
[0017] Interpolation allows input data (e.g., an input image) to be adjusted in terms of the number of elements included in the input data. With respect to the example of an input image, interpolation allows the number of pixels included in the input image to be increased. Thus, the resolution of the input image can be adjusted upward by using interpolation. It will be appreciated that this upward adjustment of the resolution will result in new data being estimated to be included in the output. Although images are described herein, it will be appreciated that the input data can be output from a previous layer of a neural network. For example, the input data can be a block of data, a tensor, etc., which is provided to a layer of a neural network (e.g., a convolutional neural network) to compute a forward pass through the layer.
[0018] Typically, interpolation is performed using integer multiples ("integer interpolation"). For example, integer interpolation may allow for doubling, tripling, quadrupling, etc., of the resolution of the input image. To perform such integer interpolation, different techniques known to those skilled in the art may be utilized. For example, nearest neighbor, bilinear, bicubic, spline, etc. Additionally, interpolation may be performed on the input image using convolution, where a window (e.g., a filter or kernel) allows for estimation of new pixels to be included in the input image. For example, bilinear interpolation may be performed using convolution, where the kernel is centered on a pixel and nearby pixels are weighted. It will be appreciated that such convolution techniques may be referred to as transposed convolution or deconvolution, for example.
[0019] Similarly, downsampling can use techniques such as averaging pixel values within a window (e.g., 2×2 pixels). In some embodiments, downsampling can therefore utilize convolution techniques (e.g., pooling, averaging of values in a window, etc.) for downsampling. Other example downsampling techniques include box sampling, mipmap, sinc, using strides, etc.
[0020] Advantageously, such convolution-based integer interpolation and optional downsampling can be efficiently implemented in a processor used to compute a forward pass of a machine learning model (e.g., a neural network model). Example processors that can be used to implement efficient convolution of input data with a window include processors (e.g., matrix processors) described in U.S. Pat. No. 11,157,287, U.S. Patent Publication No. 2019 / 0026250, and U.S. Pat. No. 11,157,441, the entire contents of which are incorporated herein by reference and constitute a part of the present disclosure, as described herein.
[0021] However, for more general interpolation needs, such integer interpolation techniques may be too rigid. As described above, autonomous or semi-autonomous vehicles may use image sensors (e.g., cameras) at least in part to obtain input data for use by machine learning models in implementing autonomous or semi-autonomous driving. These image sensors may be located around the vehicle, such as Figure 1A As shown, and thereby acquire an image for use by a machine learning model. Image sensors may output images of different resolutions, for example due to the availability of image sensors or due to the level of detail required by individual image sensors. Therefore, it may be advantageous to adjust the size (e.g., resolution) of the image from the image sensor. It will be appreciated that such adjustments may not be strictly integer-based. For example, a first image sensor may be 4 / 3 the size of a second image sensor.
[0022] As described above, fractional interpolation can allow for fractional ratio adjustment. However, implementing such fractional adjustment may generally require complex algorithms or specialized processor hardware elements. With respect to the convolution engine or processor, such fractional ratios can create technical problems because the number of pixels in each window is not fixed and the weights depend on the position of the pixels in the image. Therefore, complex hardware control may be required to ensure that fractional interpolation is implemented quickly.
[0023] In contrast, the techniques described herein allow for determining an interpolation sequence that can be efficiently implemented. In an embodiment using a convolution engine or processor, an interpolation sequence can allow for computationally efficient performance of fractional interpolation. As described above, an interpolation sequence includes one or more combinations of interpolation (e.g., upsampling) and downsampling. For example, an interpolation sequence can include two combinations. In this example, a first combination may result in interpolation of an integer factor, followed by downsampling of an integer factor. For example, interpolation may increase the resolution of an image by 3, and then downsampling may reduce the resolution of the image by 2. In this way, the input image will be effectively interpolated by a fraction of 1.5 (e.g., 3 / 2). Then, a second combination may follow the first combination. For example, the second combination may interpolate by a factor of 4, and then downsample by a factor of 3. In this example, the output of the first combination will be interpolated by a fraction of 5 / 3. Therefore, the input image of the interpolation sequence will be interpolated by a fraction of 15 / 6.
[0024] The above fractional interpolation allows efficient use of the processor through successive interpolation and downsampling rounds. The techniques described in this article can provide huge technical savings in graphics processing units (GPUs), neural processing units (NPUs), processors implementing convolution engines, hardware application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs).
[0025] block diagram
[0026] Figure 1Ais a block diagram of an example vehicle processor system 100 of an illustrated vehicle 102 that outputs an image (e.g., image N) to which fractional interpolation is applied. As shown, the vehicle includes a plurality of image sensors (e.g., cameras 104A-104N) located around the vehicle 102. The cameras 104A-104N can provide a substantially 360-degree view of the vehicle 102 so that autonomous or semi-autonomous driving can be performed. The vehicle 102 can represent a consumer vehicle, such as an electric vehicle, or can represent a commercial vehicle (e.g., a truck, a van, etc.). The vehicle 102 can also represent an unmanned vehicle, such as an unmanned aerial vehicle, an unmanned ground vehicle, etc.). In some embodiments, the functionality described herein can be implemented by a robot that includes one or more cameras.
[0027] The vehicle includes a vehicle processor system 100 having one or more processors that implements a machine learning model to autonomously or semi-autonomously drive the vehicle 102. For example, the machine learning model may include a convolutional neural network, a fully connected neural network, etc. As part of the process for implementing autonomous or semi-autonomous driving, the vehicle processor system 100 acquires images from cameras 104A-104N.
[0028] It will be appreciated that the acquired images may need to be resized (e.g., in resolution) in order to be used by the vehicle processor system 100 to implement autonomous or semi-autonomous driving. For example, cameras 104A-104N output images at different resolutions. For example, camera 104A may output images at a first resolution, while camera 104N may output images at a smaller second resolution. As another example, a machine learning model implemented by the vehicle processor system 100 may use input information at a specific resolution. For example, a machine learning model may use a vectorized image as input to an initial layer or portion of a model or an intermediate layer or subsequent portion of a model. In this example, the size of the vector may be fixed. Therefore, it may be necessary to perform fractional interpolation on the image to obtain the size of the vector.
[0029] Example images are illustrated as being provided from an image sensor (e.g., via a bus or other communication technology) to the vehicle processor system 100. Specifically, image 106A from camera 104A and image 106N from camera 104N are illustrated as being received by the vehicle processor system 100. In the illustrated embodiment, the vehicle processor system 100 fractionally interpolates image 106A to correspond to the resolution of image 106A. Thus, images 106A and 106N may be used as inputs of the same size (e.g., the same number of elements or pixels) to a machine learning model.
[0030] As will be described in more detail below, the vehicle processor system 100 can use an interpolation sequence to fractionally interpolate the image 106N at a particular fractional value. For example, the interpolation sequence can include one or more combinations of corresponding interpolation and downsampling. In this example, the number of combinations can be based on the fractional value to be applied to the image 106N. For example, if the fractional value can be achieved by a combination, the vehicle processor system 100 can perform a combination of interpolation and downsampling. An example fractional value can include 4 / 3, so that the vehicle processor system 100 interpolates the image N 106N by 4 (e.g., upsamples by 4) and downsamples by 3.
[0031] In some embodiments, the vehicle processor system 100 may limit the extent to which interpolation and downsampling are applied to the image 106N in a single combination. For example, the vehicle processor system 100 may apply interpolation up to a first threshold integer (e.g., 4, 8, 12, 32, 64, 128) and apply downsampling up to a second threshold integer (e.g., 4, 8, 12, 32, 48, 64, etc.). Thus, for certain fractional values, the vehicle processor system 100 may apply two or more combinations of interpolation and downsampling.
[0032] As an example of two or more combinations, the fractional value may be 35 / 16. For this fractional value, the vehicle processor system 100 may determine that two combinations are to be used, where the first combination represents an interpolation that results in a value of 7 / 4, and the second combination represents an interpolation that results in a value of 5 / 4. This determination may be made based on the number '35' being greater than a threshold number that the vehicle processor system 100 is configured to interpolate.
[0033] For certain score values, the vehicle processor system 100 may determine that a threshold number of combinations is to be used to achieve a value substantially close to the score value. In some embodiments, the threshold number may be two combinations. It will be appreciated that a large number of score values may be achieved using two combinations. In some embodiments, the vehicle processor system 100 applies interpolation up to a first threshold number and applies downsampling up to a second threshold number. Therefore, there may be score values that cannot be accurately achieved by two combinations. In these embodiments, the vehicle processor system 100 may determine the value that is closest to the desired score value. In some embodiments, the threshold number of combinations may be 3, 5, 7, 9. In some embodiments, there may be no threshold number of combinations.
[0034] To ensure the accuracy of the interpolation sequence, the vehicle processor system 100 may start with interpolation to increase the resolution of the image 106N. The interpolation may then be followed by downsampling. In some embodiments, the interpolation sequence may initially include two or more interpolations. In some embodiments, the interpolation may be followed by two or more downsampling.
[0035] Although Figure 1A The image 106 is illustrated as being interpolated to increase resolution, but it is understood that the techniques described herein may reduce resolution. For example, Figure 1B An example of fractional interpolation to reduce image resolution is described.
[0036] Figure 1B 1 is a block diagram of interpolating and downsampling an input image 156 according to an interpolation sequence. As shown, the interpolation sequence causes the input image 152 to be fractionally interpolated by a factor of 2 / 3. Thus, in this example, the resolution of the input image 152 is reduced.
[0037] The vehicle processor system 100 may determine the interpolation sequence based on the fractional value 2 / 3. Since this fractional value can be achieved through a combination of interpolation and downsampling, the system 100 identifies that the image 152 will be interpolated by 2 and then downsampled by 3.
[0038] In some embodiments, the vehicle processor system 100 can use convolution techniques to perform integer interpolation. In this example, a window can be moved across the pixels of the image 152, and weights can be applied to pixels near the center pixel of the window. These weights can then be used to estimate new pixels to be included in the image 152. In some embodiments, the image 152 can be interpolated according to other techniques. A convolution engine or processor can be used to perform the integer interpolation described herein. As described above, the engine or processor can include a matrix processor (e.g., a non-contracting array) that calculates the product of weights and input data (e.g., weights can be arranged on one axis of the matrix, and input data can be arranged on another axis). In some embodiments, the maximum interpolation value can be based on the size associated with the matrix.
[0039] The temporary image 154 is illustrated as being obtained by 2× interpolation. For example, the image may be stored by a convolution engine or processor (e.g., in a cache, temporary storage device, etc.). The vehicle processor 100 may then downsample the temporary image 154 by 3. For example, the temporary image 154 may be used as input data to the convolution engine or processor. Similar to the above, in some embodiments, convolution techniques may be used to perform the downsampling. For example, a window may be moved across the pixels of the image 154, and the pixels within the window may be averaged or otherwise combined to reduce the number of pixels. The resulting output 156 is shown in FIG. Figure 1B as shown, and represents a fractional interpolation of 2 / 3.
[0040] flow chart
[0041] Figure 2A4 is a flow chart of an example process 200 for performing fractional interpolation using a vehicle processor system. For convenience, process 200 will be described as being performed by a system of one or more processors (e.g., vehicle processor system 100). In some embodiments, the system may be included in a vehicle (e.g., vehicle 102, 400). In some embodiments, the system may represent a portion of a processor (e.g., a convolution engine or convolution processor).
[0042] At box 202, the system acquires an image for interpolation. As described above, the system can communicate (e.g., wired or wireless communication) with a camera used to acquire images for system processing. For example, the system can be used to enable autonomous or semi-autonomous driving of a vehicle.
[0043] At block 204, the system identifies a score value associated with interpolation of the image. The system can obtain information indicating the score value to be applied. For example, the system can implement a pipeline that pre-processes images received from a camera. In this example, the system can access information indicating the score value. For example, the system can execute code that causes the image to be interpolated according to the score value.
[0044] At block 206, the system determines an interpolation sequence based on the fractional value. As described above, the interpolation sequence can include one or more combinations of interpolation and downsampling. Thus, the interpolation sequence can include a first combination including a first interpolation followed by a second downsampling. The interpolation sequence can also include a second combination including a second interpolation followed by a second downsampling. In some embodiments, the system can limit the number of combinations to two.
[0045] To determine the interpolation sequence, the system can access information indicating corresponding integer thresholds for interpolation and downsampling. Thus, the system can limit the highest integer interpolation to 8, 16, 32, 64, 80 times. The system can also limit the highest downsampling to 8, 16, 32, 64, 80 times. These limits can be based on hardware constraints of the system, such as hardware constraints of a convolution engine or processor. These limits can also be defined in software or code being executed by the system.
[0046] With respect to the integer threshold, the system can therefore determine whether a single combination can achieve the fractional value. For example, if the fractional value is 8 / 5, the system can include a single combination in the interpolation sequence. This single combination would include interpolating the image by 8 and then downsampling the image by 5.
[0047] For fractional values that require two or more combinations, the system can determine corresponding interpolated and downsampled values for those combinations. For example, for two combinations, the system can split the fractional value into two fractions that, when multiplied, equal the fractional value. In some embodiments, the system can determine factors for the numerator and denominator. These factors can then be used to select corresponding interpolated and downsampled values. As described above, some fractional values may not be exactly achievable, and the system can select values that cause the closest interpolation to be applied.
[0048] The system applies the determined interpolation sequence at block 208. As described above, the system interpolates based on the score values using the interpolation sequence.
[0049] Figure 2B is a block diagram illustrating a convolution engine 250 of an example vehicle processor system 100 executing an interpolation sequence 252 based on fractional interpolation values 254 .
[0050] In the illustrated embodiment, the convolution engine 250 receives an image 256 and interpolates the image according to the fractional interpolation values 254. The vehicle processor system 100 or the convolution engine 250 may determine an interpolation sequence 252 for interpolation with the fractional interpolation values 254. Figure 2B As shown, the interpolation sequence 252 includes two combinations of interpolation and downsampling.
[0051] As described above, convolution engine 250 can use convolution techniques to implement interpolation. In addition, convolution engine 250 can use convolution techniques to implement downsampling. Thus, image 256 can be interpolated by engine 250, and the output is provided back to convolution engine 250 for subsequent downsampling according to interpolation sequence 252.
[0052] Figure 3 4 is a flow chart of an example process 300 for interpolating an image received from an image sensor included on an autonomous or semi-autonomous vehicle. For convenience, the process 300 will be described as being performed by a system having one or more processors (e.g., vehicle processor system 100). In some embodiments, the system may be included in a vehicle (e.g., vehicle 102, 400). In some embodiments, the system may represent a portion of a processor (e.g., a convolution engine or convolution processor).
[0053] At block 302, the system acquires an input image from an image sensor. Figure 1A As shown, the system can acquire images from cameras around the vehicle.
[0054] At box 304, the system performs fractional interpolation on at least a portion of the input image. The system identifies whether the image from each camera is to be fractionally interpolated. For example, the cameras may operate at different resolutions so that the resolution of the images may be different. In some embodiments, the system accesses the specifications of each camera to identify the resolution. In these embodiments, the system may cause the resolution of the input image to be interpolated so that it corresponds (for example, the system may use the highest resolution, the lowest resolution, the average resolution, etc.). In some embodiments, the system may implement a pre-processing pipeline that indicates that fractional interpolation is performed on images from a subset of image sensors. The system then applies a specific fractional interpolation value to at least a portion of the input image.
[0055] At block 306, the system pre-processes the input image. The system may optionally apply additional pre-processing, such as filtering, lighting adjustments, and the like.
[0056] At box 308, the system provides the input image to a machine learning model or a deep learning pipeline. Then, the image including at least some of the fractional interpolated images is provided to the machine learning model. In some embodiments, the machine learning model can be used to implement autonomous or semi-autonomous driving.
[0057] Vehicle Block Diagram
[0058] Figure 4 A block diagram of a vehicle 400 (e.g., vehicle 102) is illustrated. Vehicle 400 may include one or more motors 402 that cause vehicle 400 to move. Motor 402 may include, for example, an induction motor, a permanent magnet motor, etc. As known to those skilled in the art, battery 404 (e.g., one or more battery packs, each battery pack including a plurality of batteries) may be used to power motor 402.
[0059] Vehicle 400 also includes a propulsion system 406 that can be used to set the gear (eg, propulsion direction) of the vehicle. For an electric vehicle, propulsion system 406 can adjust the operation of electric motor 402 to change the propulsion direction.
[0060] Furthermore, the vehicle includes a vehicle processor system 100 configured to fractionally interpolate data, such as images received from image sensors (e.g., cameras 104A-104N) of the vehicle 400. The vehicle processor system 100 may additionally output information to and receive information (e.g., user input) from a display 408 included in the vehicle 400.
[0061] Other embodiments
[0062] All processes described herein can be embodied and fully automated via software code modules that are executed by a computing system including one or more computers or processors. The code modules can be stored in any type of non-transitory computer-readable medium or other computer storage device. Some or all of these methods can be embodied in dedicated computer hardware.
[0063] It will be apparent from this disclosure that there are many other variations in addition to those described herein. For example, depending on the embodiment, certain actions, events, or functions of any algorithm described herein may be performed in a different order, may be added, merged, or omitted entirely (e.g., not all described actions or events are necessary for algorithm practice). In addition, in some embodiments, actions or events may be performed concurrently, such as by multithreading, interrupt handling, or multiple processors or processor cores, or on other parallel architectures, rather than sequentially. In addition, different tasks or processes may be performed by different machines and / or computing systems that may work together.
[0064] The various illustrative logic blocks, modules, and engines described in conjunction with the embodiments disclosed herein may be implemented or executed by a machine designed to perform the functions described herein, such as a processing unit or processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The processor may be a microprocessor, but in an alternative, the processor may be a controller, a microcontroller, or a state machine, a combination thereof, or the like. The processor may include a circuit system configured to process computer executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logic operations without processing computer executable instructions. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Although primarily described herein with respect to digital technology, the processor may also primarily include analog components. For example, some or all of the signal processing algorithms described herein may be implemented in an analog circuit system or a mixed analog and digital circuit system. The computing environment may include any type of computer system, including but not limited to a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computing engine within an appliance, etc.
[0065] Unless otherwise specifically stated, conditional language such as "can," "could," "might," or "may" is understood in the context of being generally used to convey that certain embodiments include and other embodiments do not include certain features, elements, and / or steps. Thus, such conditional language generally does not indicate that one or more embodiments require the features, elements, and / or steps in any way, nor does it indicate that one or more embodiments must include logic for deciding whether to include or perform such features, elements, or steps in any particular embodiment with or without user input or prompting.
[0066] Unless specifically stated otherwise, negative language, such as the phrase "at least one of X, Y, or Z," should be understood along with the context generally used to indicate that an item, term, etc. can be X, Y, Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z are all present.
[0067] Any process description, element or block in the flowcharts described herein and / or shown in the drawings should be understood to potentially represent a code module, segment or portion, which includes one or more executable instructions for implementing a specific logical function or element in the process. Alternative implementations are included within the scope of the embodiments described herein, where elements or functions may be deleted, executed out of the order shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved.
[0068] Unless expressly stated otherwise, articles such as "a" or "an" should generally be interpreted as including one or more of the items. Thus, phrases such as "a device configured to..." are intended to include one or more of the referenced devices. Such one or more referenced devices may also be collectively configured to perform the references. For example, "a processor configured to perform references A, B, and C" may include a first processor configured to perform reference A, the first processor working in conjunction with a second processor configured to perform references B and C.
[0069] It should be emphasized that many changes and modifications may be made to the above-described embodiments, and its elements should be understood as one of other acceptable examples. All such modifications and changes are intended to be included within the scope of the present disclosure.
Claims
1. A computer-implemented method, the method comprising: acquiring images from a plurality of image sensors located around the vehicle; For a first image, determining an interpolation sequence for interpolating the first image according to fractional interpolation values, wherein the interpolation sequence comprises one or more combinations of corresponding integer interpolation and integer downsampling; and The first image is interpolated according to the interpolation sequence.
2. A method according to claim 1, wherein the interpolation sequence includes a first combination, the first combination includes a first interpolation, the first interpolation is followed by a first downsampling, and the interpolation sequence also includes a second combination receiving an output from the first combination, wherein the second combination includes a second interpolation, and the second interpolation is followed by a second downsampling. The method of claim 1 , wherein the interpolation is performed via a convolution engine or a convolution processor.
4. The method of claim 3, wherein the convolution engine or convolution processor comprises a matrix of computational elements, and wherein the matrix of computational elements is non-contractive. 5 . The method of claim 1 , wherein the interpolation of the first image comprises interpolating the first image and downsampling the first image using a convolution process.
6. The method of claim 1, wherein the integer interpolated value is multiplied by a numerator equal to the fractional interpolated value.
7. The method of claim 1, wherein the integer downsampled values are multiplied by a denominator equal to the fractional interpolated value.
8. A vehicle processor system configured to be included in a vehicle, wherein the vehicle processor system is configured to: acquiring images from a plurality of image sensors located around the vehicle; For a first image, determining an interpolation sequence for interpolating the first image according to fractional interpolation values, wherein the interpolation sequence comprises one or more combinations of corresponding integer interpolation and integer downsampling; and The first image is interpolated according to the interpolation sequence using a convolution engine of the vehicle processor system.
9. The system of claim 8, wherein the interpolation sequence comprises a first combination comprising a first interpolation followed by a first downsampling, and the interpolation sequence further comprises a second combination receiving an output from the first combination, wherein the second combination comprises a second interpolation followed by a second downsampling.
10. The system of claim 8, wherein the interpolation is performed via a convolution engine or a convolution processor.
11. The system of claim 10, wherein the convolution engine or convolution processor comprises a matrix of computational elements, and wherein the matrix of computational elements is non-contractive. 12 . The system of claim 8 , wherein the interpolation of the first image comprises interpolating the first image and downsampling the first image using a convolution process.
13. The system of claim 7, wherein the integer interpolated value is multiplied by a numerator equal to the fractional interpolated value.
14. The system of claim 8, wherein the integer downsampled values are multiplied by a denominator equal to the fractional interpolated value.
15. A non-transitory computer storage medium storing instructions that, when executed by a system of one or more processors, cause the one or more processors to: acquiring images from a plurality of image sensors located around the vehicle; For a first image, determining an interpolation sequence for interpolating the first image according to fractional interpolation values, wherein the interpolation sequence comprises one or more combinations of corresponding integer interpolation and integer downsampling; and The first image is interpolated according to the interpolation sequence using a convolution engine of the vehicle processor system.
16. The computer storage medium of claim 15, wherein the interpolation sequence comprises a first combination comprising a first interpolation followed by a first downsampling, and the interpolation sequence further comprises a second combination receiving an output from the first combination, wherein the second combination comprises a second interpolation followed by a second downsampling.
17. The computer storage medium of claim 15, wherein interpolation is performed via a convolution engine or a convolution processor.
18. The computer storage medium of claim 15, wherein the interpolation of the first image comprises interpolating the first image and downsampling the first image using a convolution process.
19. The computer storage medium of claim 15, wherein the integer interpolated value is multiplied by a numerator equal to the fractional interpolated value.
20. The computer storage medium of claim 15, wherein the integer downsampled values are multiplied by a denominator equal to the fractional interpolated value.
