Image processing device, image processing method, and image processing program

Abstract

The objective is to provide a video processing device, a video processing method, and a video processing program that enable high-quality and low-latency video conversion processing by performing highly parallel processing on a block-by-block basis while reducing access to DRAM. [Solution] The system includes an input image storage means that stores the input video as a pre-conversion frame image, a block storage means that stores the pixel data of a predetermined range of the pre-conversion frame image read from the input image storage means and is configured as a memory that allows random access to the pixel data, an interpolation means that calculates the converted pixel data from the pre-conversion pixel data by a predetermined interpolation operation, an output image storage means that stores the converted frame image consisting of the converted pixel data, and a video output means that outputs the converted frame image as an output video. The block storage means reads the image data by referring to the pixels in the pre-conversion frame image calculated by performing back projection on each pixel of a unit block formed by dividing the converted frame image into predetermined sizes, and the interpolation means reads the pixel data of pixels located around the pre-conversion pixels corresponding to the converted pixels from the block storage means by referring to the pixel data of the pixels located around the pre-conversion pixels, and calculates the converted pixel data by an appropriate interpolation operation method.

Description

Technical Field

[0001] The present invention relates to a video processing apparatus, a video processing method, and a video processing program capable of converting an image acquired by a fisheye camera into an equirectangular projection (equirectangular format) with high resolution and low latency.

Background Art

[0002] In recent years, VR services that allow users to experience the situation in a remote location as if they were on site using head-mounted displays, dome displays, etc. have become widespread. Live-action panoramic video for VR is generated by arranging a plurality of cameras using fisheye lenses, flattening the video captured by each camera by the equirectangular projection method or the like for each frame image, and then combining the same regions pixel by pixel.

[0003] Since the lens of a fisheye camera is a fisheye lens, the video acquired by the fisheye camera causes a large distortion toward the upper, lower, left, and right ends. As it is, a three-dimensional effect or depth cannot be obtained even when viewed by humans, and there is also a problem that objects cannot be correctly recognized even in image recognition by AI.

[0004] To solve this problem, by going through a processing step of converting the video into an equirectangular format video by the equirectangular projection method, the direction and angle can be determined from the converted video, and thus it can be applied to VR video.

[0005] In the equirectangular projection method, it is necessary to interpolate the pixels after conversion using the pixels before conversion. As a method for this interpolation process, a method of performing bilinear interpolation or the like using a plurality of pixels before conversion is disclosed (see Patent Document 1).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] However, when converting a fisheye image to an equirectangular image using equirectangular projection, conventional methods such as bilinear interpolation require reading multiple pixel data points from the input image acquired through the fisheye lens before conversion.

[0008] In particular, DRAM is commonly used as a storage method for input video, where data is read out one line at a time. However, in order to perform bilinear interpolation, it is necessary to refer to the pixel data of four pixels (2x2 pixels) from the DRAM when calculating the pixel data of one pixel after conversion. As a result, one line is read out for each pixel, which has led to the problem that conventional methods cannot perform interpolation calculations at high speed.

[0009] Furthermore, converting images captured with a fisheye lens to equirectangular projection images requires, according to conventional methods, post-capture software conversion. This conversion process must account for the circumferential distortion of the fisheye lens, resulting in the constant use of transcendental functions (sin / cos, etc.), which generates a large amount of computational processing that software is particularly poor at. This poses significant challenges to real-time performance and processing latency, especially when converting high-resolution images exceeding 4K (4K UHDTV).

[0010] Furthermore, in the conversion process, simply performing a coordinate transformation leaves unnaturalness in the converted image. To correct this, interpolation processing such as bilinear interpolation, which uses the average value of at least four points, is required, which quadruples the amount of computation and further contributes to delays.

[0011] Fisheye cameras offer a wide field of view and mathematically guaranteed orientation and angle, making them promising for applications in various fields, including VR services, autonomous driving using in-vehicle cameras, ADAS, monocular SLAM, backup cameras, remote control of construction machinery and robotic arms, pipe inspection, surveillance cameras with minimal blind spots, and sports judging cameras. However, these applications demand high resolution and real-time performance, necessitating the resolution of delays in the conversion process required to correct these distortions.

[0012] This invention has been made in view of the above problems, and aims to provide an image processing device, an image processing method, and an image processing program that can perform image conversion processing with high image quality and low latency by performing highly parallel processing for each block while reducing access to DRAM. [Means for solving the problem]

[0013] To solve the above problems, the present invention provides an image processing device that processes an image from an input image composed of a fisheye image to an output image composed of an equirectangular image by equirectangular transformation accompanied by a predetermined interpolation operation, comprising: an input image storage means for storing the input image as a pre-transformation frame image; a block storage means configured as a memory that stores pixel data of a predetermined range of the pre-transformation frame image read from the input image storage means and allows random access to the pixel data; an interpolation means for calculating post-transformation pixel data from the pre-transformation pixel data by a predetermined interpolation operation; an output image storage means for storing a post-transformation frame image consisting of the post-transformation pixel data; and an image output means for outputting the post-transformation frame image as an output image, wherein the block storage means reads out image data by referring to pixels in the pre-transformation frame image calculated by performing back projection on each pixel of a unit block formed by dividing the post-transformation frame image into predetermined sizes, and the interpolation means reads out pixel data from the block storage means by referring to pixel data of pixels located around the pre-transformation pixels corresponding to the post-transformation pixels, and calculates post-transformation pixel data by an appropriate interpolation operation method.

[0014] The block storage means in the image processing apparatus of the present invention can also be configured to classify the converted frame image into parallel blocks consisting of a plurality of unit blocks, and to perform multi-parallel processing for each classification.

[0015] Furthermore, the interpolation means may also be configured to refer to the unconverted pixels corresponding to all pixels included in the unit block, read the pixel data of pixels located in a predetermined region with the minimum value of the reference coordinates as the origin from the input image storage means to the block storage means, and perform an interpolation operation.

[0016] Furthermore, the back projection, with respect to the pixel coordinates (x,y) of the converted frame image, The following formula: TIFF2026076878000001.tif33136(here, [1] TIFF2026076878000002.tif5094 (R represents the radius of the virtual sphere, α represents the longitude of point P on the virtual sphere corresponding to the pixel at coordinates (x,y), and β represents the latitude.) [2] TIFF2026076878000003.tif23169(u represents the x-coordinate of point P1 projected onto a virtual sphere of radius 1, v represents the y-coordinate, and w represents the z-coordinate.) [3] TIFF2026076878000004.tif19152 (r is the distance from point P1 to the center of the image circle, K0, K1, K2...K6 are the radial distortion coefficients in the fisheye lens, and θ represents the angle of incidence at point P1.) (Let's assume that.) This can also be done by calculating the pixel coordinates (u',v') of the pre-conversion frame image.

[0017] Furthermore, the video processing device can be operated using a video processing program that enables the computer to function as each of the aforementioned means. [Effects of the Invention]

[0018] As described above, according to the present invention, by redesigning the internal circuit configuration of the FPGA without changing the basic calculation algorithm of the imaging device, it becomes possible to convert fisheye images to equirectangular images in near real-time with a delay of 16.7ms or less, even for high-resolution video exceeding 4K, with interpolation. This makes it possible to apply the technology not only to VR services but also to various fields such as autonomous driving using in-vehicle cameras, ADAS, monocular SLAM, backup cameras, remote control of construction machinery and robotic arms, pipe inspection, surveillance cameras with minimal blind spots, and sports judgment cameras. [Modes for carrying out the invention]

[0019] Embodiments of the present invention will be described with reference to diagrams and mathematical formulas. Needless to say, the technical scope of the present invention is not limited to the following embodiments.

[0020] The video processing apparatus 1 of the present invention converts an input video into an output video by conversion processing involving an interpolation step.

[0021] In this embodiment, the video processing apparatus 1 is configured to convert an input video captured by a fisheye camera into an output video in the equirectangular format as a method for converting a video involving interpolation.

[0022] As shown in the following table, the video processing apparatus 1 includes an input video acquisition unit A, an input image storage unit Mi, a memory block generation unit Bg, a block storage unit Mb, an interpolation unit Ip, an output image storage unit Mo, and a video output unit O. [Table 1]

[0023] The input video acquisition unit A acquires video information projected onto an image sensor through a fisheye lens, and here, acquires a video composed of a fisheye image G1.

[0024] The input video acquisition unit A sequentially writes and stores the fisheye image G1, which is a frame of the input video, in the input image storage unit Mi.

[0025] Further, when the writing of one frame to the input image storage unit Mi is completed, the input video acquisition unit A notifies the memory block generation unit Bg of the completion of writing.

[0026] The input image storage unit Mi stores the frame images constituting the input video, and here, stores the fisheye image G1 acquired by the input video acquisition unit A.

[0027] The input image storage unit Mi can be composed of a semiconductor memory, and for example, can be composed of a general DRAM as an image memory.

[0028] The fisheye image G1 stored in the input image storage means Mi is read out by the memory block generation means Bg.

[0029] The memory block generation means Bg divides the frame image stored in the input image storage means Mi into memory blocks of predetermined size in the horizontal and vertical directions, each larger than the number of pixels required for interpolation calculations of all pixels contained in the unit block after conversion by the conversion process. Each memory block is sized to contain at least the pixels required for interpolation calculations by the interpolation means Ip described later, for example, 4 pixels (2x2 pixels) in the case of bilinear interpolation.

[0030] [Table 2]

[0031] For example, in the 4K panoramic version, the output video is divided into 240[pixel] x 216[line] unit blocks, and the range of the input video referenced by all pixels in each unit block is read from the input image storage means Mi to the block storage means Mb. Then, conversion and interpolation processing are performed while randomly accessing that pixel data.

[0032] At this time, in order to write pixel data from the input image storage means Mi to the block storage means Mb, the coordinates of the reference destination of the input image are calculated from all the coordinates contained in the unit block.

[0033] The origin of the pixel data to be written to the block storage means Mb is determined from the minimum value of the calculated reference coordinates, and in this embodiment, 384 [pixel] x 384 [line] pixel data based on that origin is written to the block storage means Mb.

[0034] The pixel data of the four neighboring points of the calculated reference coordinates is read from the block storage means Mb, and bilinear interpolation is performed to generate the output pixel data.

[0035] In this embodiment, the 4K panoramic shooting equipment (SONY α6400 + ENTANIYA Fisheye HAL 200) used has a Raw Image Buffer size of 384 pixels x 384 lines. This allows it to cover 240 pixels x 216 lines of reference pixel data even at the edges where distortion is greatest. However, if the radius value is even smaller and the distortion is even greater, it is necessary to either reduce the block size or increase the size of the Raw Image Buffer.

[0036] [Table 3]

[0037] When the memory block generation means Bg has finished writing the memory block to the block storage means Mb, it notifies the interpolation means Ip that the writing is complete.

[0038] The block storage means Mb stores the memory blocks generated by the memory block generation means Bg, and is a memory that can be randomly accessed on a pixel-by-pixel basis. In this embodiment, it is composed of an FPGA block RAM.

[0039] The block storage means Mb arranges multiple block RAMs, for example, two blocks horizontally and one block vertically, and sequentially updates the memory blocks.

[0040] Furthermore, if the block RAM has two output ports, only two pixel data points can be read from the memory block at a time. In this case, the memory block generation means Bg writes the same memory block twice to the block storage means Mb, and the interpolation means Ip reads two pixels from each memory block, for a total of four pixel data points.

[0041] The interpolation means Ip reads pixel data of surrounding pixels at the original pixel position corresponding to the converted pixel position from the memory block, and calculates the pixel data of the converted pixel position through interpolation.

[0042] Here, the pixel data of the equirectangular image G2 is interpolated and generated by referring to the memory blocks stored in the block storage means Mb.

[0043] The interpolation means Ip comprises a pre-transformation coordinate calculation means Ic and an interpolation pixel data calculation means Id.

[0044] The pre-transformation coordinate calculation means Ic calculates the pre-transformation pixel position (coordinate) that corresponds to the post-transformation pixel position.

[0045] Here, the pre-transformation coordinate calculation means Ic calculates the position (coordinate) of the pixels in the pre-transformation fisheye image G1 that correspond to the pixels in the transformed equirectangular image G2.

[0046] In other words, the pre-conversion coordinate calculation means Ic calculates the pixel positions (coordinates P'(u',v')) of the fisheye image G1 that correspond to the pixel positions (coordinates P(x,y)) of the equirectangular image G2 when the fisheye image G1 of (a) is converted to an equirectangular image G2, as shown in the table below. [Table 4]

[0047] In this embodiment, the conversion from the fisheye image G1 to the equirectangular image G2 is performed using a general equirectangular transform. Therefore, the pre-conversion coordinate calculation means Ic can calculate the pixel positions (coordinates P'(u',v')) of the fisheye image G1 before conversion, which correspond to the pixel positions (coordinates P(x,y)) of the equirectangular image G2, by performing back projection (inverse transform) of the equirectangular transform.

[0048] In other words, if we let R be the radius of the image circle with a field of view of 180° captured by a fisheye lens, and convert this image circle to an equirectangular projection, we get the situation shown in the table below. [Table 5]

[0049] Here, the longitude α and latitude β on a virtual sphere of radius R for the coordinates (x,y) of any point P on an equirectangular projection can be expressed using R as follows:

number

[0050] Furthermore, the coordinates (u,v,w) of point P1, which is projected onto a virtual sphere of radius 1, are as follows: [Table 6]

[0051] The coordinates (u,v,w) of point P1 can be expressed using longitude α and latitude β as follows:

number

[0052] Furthermore, if we let θ be the angle of incidence at point P1, that is, the angle between the line connecting the center O of the virtual sphere and point P1 and the z-axis, then θ can be expressed by the following equation. [Table 7]

[0053] When light incident through a fisheye lens is projected onto an image sensor, the distance r between the projected point and the center of the image circle can be expressed as a function of the angle of incidence θ.

[0054] In the fisheye lens used in this embodiment, the distance r between the projected point and the center of the image circle can be approximated using a polynomial of the angle of incidence θ, for example, as follows:

number

[0055] When performing calibration that takes into account the distortion of a fisheye lens through the above calculation process, the coordinates (u',v') of the point projected onto the xy-plane from point P can be expressed by the following equation.

number

[0056] In the image processing method of the present invention, pixels of the input image (u',v') are read out and processed by back projection from the output coordinates (x,y). However, as mentioned above, since random access is not possible with DRAM, processing using a raster scan method increases redundant memory access, resulting in significantly reduced processing efficiency.

[0057] For example, in the 4K panoramic version, the output video is divided into 240[pixel] x 216[line] blocks, and the range of the input video referenced by each block is extracted from the input image storage means Mi to the block storage means Mb, where it is processed while randomly accessing it.

[0058] Next, in order to write pixel data from the input image storage means Mi to the block storage means Mb, the reference coordinates of the input image are calculated from all the coordinates contained in the block using the method described above.

[0059] The origin of the pixel data to be written to the block storage means Mb is determined from the minimum value of the calculated reference coordinates, and 384 [pixel] x 384 [line] pixel data is written to the block storage means Mb based on that origin.

[0060] The pixel data of the four neighboring points of the calculated reference coordinates is read from the block storage means Mb, and bilinear interpolation is performed to generate the output pixel data.

[0061] The coordinate calculations and pixel data reading for a 240[pixel] x 216[line] block will be performed in four parallel operations, with each block being further divided into four 240[pixel] x 54[line] sections.

[0062] In this regard, when 4K output video is divided into 240[pixel] x 216[line] blocks, it can be divided into 16 horizontally and 10 vertically, for a total of 160 blocks. In this embodiment, in order to achieve 60fps in the 4K panoramic corrected version, the 4K video is divided into three sections of 60 blocks each (6 horizontally and 10 vertically), and each divided area is processed simultaneously.

[0063] On the right side, a total of 60 blocks are processed (6 horizontally and 10 vertically), but only the area of ​​4 horizontally and 10 vertically will be displayed on the output screen.

[0064] The pre-transformation coordinate calculation means Ic outputs the transformed pixel position (coordinate p) and the pre-transformation pixel position (coordinate p') to the interpolated pixel data calculation means Id.

[0065] The interpolated pixel data calculation means Id calculates the pixel data of the pixel position before transformation, calculated by the pre-transformation coordinate calculation means Ic, using the pixel data of surrounding pixels by interpolation.

[0066] Here, the interpolated pixel data calculation means Id calculates pixel data using bilinear interpolation. It reads the pixel data of the four surrounding pixels of the pixel position before conversion from a memory block stored in the block storage means Mb, and calculates the pixel data using bilinear interpolation. This calculated pixel data becomes the pixel data of the converted pixel position.

[0067] Specifically, as shown in the table below, the interpolated pixel data calculation means Id assumes that the coordinates p' of the pixel position before transformation are (x', y'), and the coordinates of the four surrounding pixels are p1([x'],[y']), p2([x']+1,[y']), p3([x'],[y']+1), and p4([x']+1,[y']+1). Note that [x'] and [y'] represent the integer parts of x' and y', respectively. [Table 8]

[0068] The interpolated pixel data calculation means Id reads the pixel data for coordinates p1, p2, p3, and p4 from the memory blocks corresponding to coordinates p1, p2, p3, and p4, and calculates the pixel data for coordinate p' using the following equation 5 through bilinear interpolation.

[0069]

number

[0070] Here, let Dst(x',y') be the pixel data at coordinate p', Src([x'],[y']) be the pixel data at coordinate p1, Src([x']+1,[y']) be the pixel data at coordinate p2, Src([x'],[y']+1) be the pixel data at coordinate p3, and Src([x']+1,[y']+1) be the pixel data at coordinate p4.

[0071] The interpolated pixel data calculation means Id writes the calculated pixel data to the output image storage means Mo as pixel data for the pixel position (coordinate p) of the converted equirectangular image G2.

[0072] Furthermore, it is preferable that the interpolated pixel data calculation means Id writes the pixel data for one line of the equirectangular image G2 to the output image storage means Mo each time it calculates the pixel data for one horizontal line of the equirectangular image G2.

[0073] When the interpolation means Ip reaches the final line of the pixel referenced in the memory block, it instructs the memory block generation means Bg to generate the next memory block.

[0074] Furthermore, after the interpolation of the equirectangular image G2 is completed, the interpolation means Ip notifies the video output means O that writing to the output image storage means Mo is complete.

[0075] The output image storage means Mo stores the pixel data of the converted pixel positions calculated by the interpolation means Ip as the pixel data of the corresponding pixel positions in the frame images that constitute the output video, and the equirectangular image G2 is stored as the converted image.

[0076] The output image storage means Mo can be made of semiconductor memory, for example, DRAM, which is commonly used as image memory.

[0077] The equirectangular image G2 stored in the output image storage means Mo is read out by the video output means O.

[0078] The video output means O outputs an output video in which the image (equirectangular image G2) stored in the output image storage means Mo is used as the frame image.

[0079] The video output means O outputs the equirectangular image G2 as a frame image at the timing when it is notified by the interpolation means Ip that the writing of the equirectangular image G2 has been completed.

[0080] As described above, the image processing device 1 of the present invention can generate high-quality images in near real-time while performing bilinear interpolation calculations when converting a fisheye image to an equirectangular image.

[0081] Furthermore, when performing interpolation calculations, the image processing device 1 can read multiple pixel data before conversion from a memory block that allows reading of each pixel individually. This allows the image processing device 1 to access the pixels to be referenced during interpolation calculations on a pixel-by-pixel basis at once. Compared to methods that read one pixel per line, such as DRAM, this enables the conversion of images captured by a fisheye camera into equirectangular images at high speed and with high image quality.

[0082] The video processing device 1 can be operated by a program (video processing program) that enables each of the aforementioned means to function.

[0083] The image processing method according to the embodiment of the present invention can be broadly divided into the following four steps, A to D. A: Read pixel data from input image storage means Mi and write it to block storage means Mb. i: Calculate the reference coordinates from the output coordinates using back projection. U: Read pixel data from block storage means Mb and packetize the output pixel data along with position information, etc. E: Write pixel data to the output image storage means Mo.

[0084] The processing steps A through D are repeated, but the processing starts with step A, and once step A is completed, steps B, C, and D begin. When step C is completed, even if step D is still in progress, step A will begin for the next block. [Table 9]

[0085] First, in step S1, the input video acquisition means A acquires video using the fisheye image G1 as a frame and sequentially writes and stores it in the input image storage means Mi.

[0086] In step S2, the memory block generation means Bg stores in the block storage means Mb memory blocks obtained by dividing the pre-conversion pixels corresponding to the pixels in the unit blocks of the converted equirectangular image G2 from the fisheye image G1 stored in the input image storage means Mi in step S1 into predetermined numbers of pixels in the horizontal and vertical directions.

[0087] At this time, the memory block generation means Bg reads pixel data for a number of lines corresponding to the number of pixels in the vertical direction of the memory block from the input image storage means Mi line by line, and writes it as individual memory blocks in the block storage means Mb for each number of pixels in the horizontal direction of the memory block.

[0088] In step S3, the pre-transformation coordinate calculation means Ic of the interpolation means Ip calculates the pixel positions (coordinates) of the fisheye image G1 before equirectangular transformation that correspond to the pixel positions of the equirectangular image G2.

[0089] Here, the pre-transformation coordinate calculation means Ic calculates the pixel positions (coordinates) of the fisheye image G1 by performing equirectangular transformation back projection onto the pixel positions of the equirectangular image G2.

[0090] In step S4, the interpolated pixel data calculation means Id of the interpolation means Ip obtains pixel data of the four surrounding pixels of the pixel position (coordinates) of the fisheye image G1 calculated in step S3 from the corresponding memory block stored in the block storage means Mb.

[0091] In step S5, the interpolated pixel data calculation means Id calculates the pixel data of the equirectangular image G2 after equirectangular transformation by bilinear interpolation calculation from the pixel positions (coordinates) and respective pixel data of the four pixels acquired in step S4.

[0092] In step S6, the interpolation means Ip writes and stores the pixel data of the equirectangular image G2 calculated in step S5 into the output image storage means Mo.

[0093] The interpolation means Ip updates the memory blocks sequentially by instructing the memory block generation means Bg to generate the next memory block when the pixels referenced in the memory block reach the final line.

[0094] In step S7, the interpolation means Ip determines whether the calculation of pixel data has been completed for all pixels of the equirectangular image G2.

[0095] If the calculation of pixel data has not yet been completed for all pixels (No in step S7), the interpolation means Ip returns to step S3 and continues its operation. On the other hand, if the calculation of pixel data has been completed for all pixels (Yes in step S7), the video output means O outputs a video in step S8 using the equirectangular image G2 stored in the output image storage means Mo as the frame.

[0096] In step S9, the input video acquisition means A determines whether or not there is input for the next frame.

[0097] If the next frame is input (Yes in step S9), the video processing device 1 returns to step S1 and continues its operation. On the other hand, if the next frame is not input (No in step S9), the video processing device 1 terminates its operation.

[0098] Through the above operating process, the image processing device 1 can convert images captured by a fisheye camera into equirectangular images at high speed and with high image quality, compared to methods such as DRAM that read one pixel per line.

[0099] In this embodiment, the fisheye image G1 is described as having a configuration (RGB, etc.) where pixel data corresponds to the pixel position. However, it goes without saying that it may also be an image in which the chromatic difference component is downsampled from the luminance component (Y) and chromatic difference components (Cb,Cr), such as 4:2:0 or 4:2:2 of YCbCr.

[0100] Furthermore, the interpolation operation in the interpolation means Ip may be, for example, a bicubic interpolation operation. When using a bicubic interpolation operation, the memory block generation means Bg only needs to generate a memory block of a size that includes at least 16 pixels (4x4 pixels) referenced for the interpolation operation in one block. The interpolated pixel data calculation means Id of the interpolation means Ip then reads the pixel data of the 16 pixels surrounding the pixel position before conversion from the memory block stored in the block storage means Mb, and calculates the pixel data using a bicubic interpolation operation.

[0101] In this embodiment, the image processing device 1 is described as converting an image captured by a fisheye camera into an equirectangular projection image. However, the image processing device 1 can convert any type of image as long as it involves interpolation. For example, it can be configured to enlarge an image by pixel interpolation (inputting a 4K image and outputting an 8K image). In this case, the image processing device 1 simply inputs a 4K image as a frame image, performs enlargement by interpolation, and then outputs an 8K image as a frame image. The pre-conversion coordinate calculation means Ic of the interpolation means Ip calculates the pre-enlargement coordinates from the enlarged coordinates.

[0102] Furthermore, the present invention comprehensively encompasses the following: [1] An image processing apparatus that processes an image from an input image composed of a fisheye image to an output image composed of an equirectangular image by equirectangular transformation accompanied by a predetermined interpolation operation, comprising: an input image storage means for storing the input image as a pre-transformation frame image; a block storage means configured as a memory that stores pixel data of a predetermined range of the pre-transformation frame image read from the input image storage means and allows random access to the pixel data; an interpolation means for calculating post-transformation pixel data from the pre-transformation pixel data by a predetermined interpolation operation; an output image storage means for storing a post-transformation frame image consisting of the post-transformation pixel data; and an image output means for outputting the post-transformation frame image as an output image, wherein the block storage means reads out image data by referring to pixels in the pre-transformation frame image calculated by performing back projection on each pixel of a unit block formed by dividing the post-transformation frame image into predetermined sizes; and the interpolation means reads out pixel data from the block storage means by referring to pixel data of pixels located around the pre-transformation pixels corresponding to the post-transformation pixels, and calculates post-transformation pixel data by an appropriate interpolation operation method. [2] The block storage means classifies the converted frame image into parallel blocks consisting of a plurality of the unit blocks, and performs multi-parallel processing for each classification, as described in [1]. [3] The interpolation means reads pixel data of pixels located in a predetermined region with the origin set at the minimum value of the reference coordinates from the input image storage means to the block storage means and performs an interpolation operation by referring to the unconverted pixels corresponding to all pixels included in the unit block. [1] or [2] [4] The back projection is performed with respect to the coordinates (x,y) of the pixels in the converted frame image, The following formula: TIFF2026076878000019.tif33136(here, [1] TIFF2026076878000020.tif5094 (R represents the radius of the virtual sphere, α represents the longitude of point P on the virtual sphere corresponding to the pixel at coordinates (x,y), and β represents the latitude.) [2] TIFF2026076878000021.tif23169(u represents the x-coordinate of point P1 projected onto a virtual sphere of radius 1, v represents the y-coordinate, and w represents the z-coordinate.) [3] TIFF2026076878000022.tif19152 (r is the distance from point P1 to the center of the image circle, K0, K1, K2...K6 are the radial distortion coefficients in the fisheye lens, and θ represents the angle of incidence at point P1.) (Let's assume that.) The image processing device described in any of [1] to [3] is performed by calculating the coordinates (u',v') of the pixels of the pre-conversion frame image. [5] A video processing program for causing a computer to function as a video processing device as described in any of [1] to [4]. [Industrial applicability]

[0103] As described above, the present invention, by redesigning the internal circuit configuration of the FPGA without changing the basic calculation algorithm of the imaging device, etc., enables near real-time conversion processing from fisheye images to equirectangular images with interpolation, even for high-resolution images exceeding 4K, with a delay of 16.7ms or less. This makes it possible to apply the invention not only to VR services but also to various fields such as autonomous driving using in-vehicle cameras, ADAS, monocular SLAM, backup cameras, remote control of construction machinery and robot arms, pipe inspection, surveillance cameras with minimal blind spots, and sports judgment cameras. Therefore, it is extremely suitable for application to image processing devices, image processing methods, and image processing programs.

Claims

1. An image processing device that processes an image from an input image composed of a fisheye image to an output image composed of an equirectangular image by equirectangular transformation accompanied by a predetermined interpolation operation, An input image storage means that stores the input video as a pre-conversion frame image, A block storage means is configured to store pixel data of a pre-conversion frame image within a predetermined range read from the input image storage means, and to allow random access to the said pixel data. An interpolation means that calculates converted pixel data from pre-conversion pixel data using a predetermined interpolation operation, An output image storage means that stores a converted frame image consisting of converted pixel data, It has a video output means that outputs the converted frame image as output video, The block storage means reads out the image data by referring to the pixels in the pre-conversion frame image, which are calculated by performing back projection on each pixel of a unit block obtained by dividing the converted frame image into predetermined sizes. The interpolation means reads pixel data of pixels located around the original pixels corresponding to the converted pixels from the block storage means by referring to the pixel data of the original pixels and calculates the converted pixel data by an appropriate interpolation calculation method.

2. The image processing apparatus according to claim 1, wherein the block storage means classifies the converted frame image into parallel blocks consisting of a plurality of the unit blocks, and performs multi-parallel processing for each classification.

3. The image processing apparatus according to claim 1 or 2, wherein the interpolation means refers to the unconverted pixels corresponding to all pixels included in the unit block, reads the pixel data of pixels located in a predetermined region with the minimum value of the reference coordinates as the origin from the input image storage means to the block storage means and performs an interpolation operation.

4. The aforementioned back projection, with respect to the coordinates (x, y) of the pixels in the converted frame image, The following formula: The image processing apparatus according to any one of claims 1 to 3, which is performed by calculating the coordinates (u', v') of the pixels of the pre-conversion frame image. (Here, 〔１〕 (R represents the radius of the virtual sphere, α represents the longitude of point P on that virtual sphere corresponding to the pixel at coordinates (x, y), and β represents the latitude.) 〔２〕 (u is the point P projected onto a virtual sphere of radius 1) １ (The x-coordinate of 'x', the y-coordinate of 'v', and the z-coordinate of 'w' are shown.) 〔３〕 (r is point P) 1 and distance from the center of the image circle, K ０ _K １ _K ２ ...K ６ θ is the radial distortion coefficient in the fisheye lens, and θ is point P. １ (This represents the angle of incidence.) (Let's assume that.)

5. A video processing program for causing a computer to function as a video processing device according to any one of claims 1 to 4.

Images