Image processing method and system for vehicle surround view system

Abstract

The present invention provides an image processing method and system for a vehicle surround view system, the method comprising: receiving a low frame rate image sequence output by a camera of the vehicle surround view system; calculating a motion vector of a same pixel block on two adjacent frames of images; correcting the motion vector according to camera parameters and vehicle motion parameters; performing distortion correction on each of two adjacent frames of images based on the camera parameters; inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence. The present invention takes into account the influence of the camera's internal and external parameters and the vehicle's motion parameters on motion estimation and motion compensation, so that the generated intermediate frames are more realistic, and a high frame rate panoramic video is obtained, which improves the user's viewing experience compared to traditional panoramic systems.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to an image processing method and system for a vehicle surround view system. Background Art

[0002] With the development of technology, the maximum refresh rate supported by display screens is getting higher and higher, far greater than the mainstream video frame rate. Video dynamic interpolation technology has appeared on mobile phones and TVs. The main implementation method is to perform motion estimation and motion compensation on the video image sequence and insert an intermediate frame between two frames. This can increase the video frame rate, reduce ghosting, and make the viewing experience smoother. The current motion estimation and motion compensation algorithms do not take into account the situation of large image distortion, while the image of the vehicle's surround view system is a distorted image, and the vehicle's own motion will also affect the traditional video dynamic interpolation technology. Summary of the invention

[0003] The present invention aims at the technical problems existing in the prior art and provides an image processing method and system for a vehicle surround view system.

[0004] According to a first aspect of the present invention, there is provided an image processing method for a vehicle surround view system, comprising: receiving a low frame rate image sequence output by a camera of the vehicle surround view system; calculating a motion vector of a same pixel block on two adjacent frames of images; correcting the motion vector according to camera parameters and vehicle motion parameters; performing distortion correction on each of two adjacent frames of images based on the camera parameters; and inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence.

[0005] Based on the above technical solution, the present invention can also make the following improvements.

[0006] Optionally, the calculation of the motion vector of the same pixel block on two adjacent image frames includes: dividing each image frame into multiple pixel blocks, searching for matching pixel blocks on two adjacent image frames, the matching pixel blocks on two adjacent image frames are the same pixel block on the two adjacent image frames; and obtaining the relative displacement between the matching pixel blocks on the two adjacent image frames as the motion vector.

[0007] Optionally, correcting the motion vector according to the camera parameters and the self-vehicle motion parameters includes: calibrating the camera's intrinsic parameters and distortion parameters based on the Zhang Zhengyou calibration method, and obtaining the self-vehicle motion parameters; correcting the motion vector according to the camera's intrinsic parameters, distortion parameters and the self-vehicle motion parameters.

[0008] Optionally, the correcting the motion vector according to the camera's intrinsic parameters, distortion parameters, and the vehicle's motion parameters includes:

[0009]

[0010]

[0011] Among them, (Δx, Δy) is the motion vector of the pixel block along the x and y axes, f x 、f y is the camera internal parameter, θ d , r is determined according to the camera distortion parameters and the coordinates of the pixel block, V is the longitudinal velocity of the vehicle, t is the time interval between two adjacent frames, and α and β are calibrable coefficients.

[0012] Optionally, the performing distortion correction on each of two adjacent frames of images based on camera parameters includes:

[0013] According to the Kannala-brandt distortion projection model, the conversion relationship between the image pixels before distortion correction and the image pixels after distortion correction is established:

[0014]

[0015]

[0016] r 2 =x 2 +y 2 ; θ = arctan(r);

[0017] θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 );

[0018] Where (x′, y′) is the pixel coordinate of the image after fisheye distortion correction, (x, y) is the pixel coordinate of the distorted image output by the camera, and k1, k2, k3, k4 are the camera distortion parameters;

[0019] Based on the camera intrinsic parameters, the pixel coordinates of the image after distortion correction are converted to the ideal image pixel coordinates:

[0020] u=f x x′+C x ;

[0021] v=f y y′+C y ;

[0022] In the formula, f x 、f y , C x , C y is the camera internal parameter, (u, v) is the ideal image pixel coordinate.

[0023] Optionally, inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence includes:

[0024] According to the unidirectional motion estimation method, N frames of images are inserted between two adjacent frames of images, and the inserted intermediate frames of images are numbered 1, 2, ..., i, .., N;

[0025] Among them, the pixel coordinates of each inserted intermediate frame image are:

[0026]

[0027] Wherein, (u, v) is the pixel coordinate of the previous frame image in the previous and next two frames, Δu′ and Δv′ are the corrected motion vectors; multiple intermediate frames are inserted between every two frames in the low frame rate image sequence to obtain a high frame rate image sequence.

[0028] Optionally, the receiving of the low frame rate image sequence output by the camera of the vehicle surround view system includes: receiving the low frame rate image sequence output by multiple cameras of the vehicle surround view system; performing frame insertion processing on each low frame rate image sequence to obtain each high frame rate image sequence; and performing splicing and illumination uniformization processing on each high frame rate image sequence to obtain a high frame rate panoramic video.

[0029] According to a second aspect of the present invention, there is provided an image processing system for a vehicle surround view system, comprising: a receiving module for receiving a low frame rate image sequence output by a camera of the vehicle surround view system; a calculating module for calculating a motion vector of a same pixel block on two adjacent frames of images; a correcting module for correcting the motion vector according to camera parameters and vehicle motion parameters; and performing distortion correction on each of two adjacent frames of images based on the camera parameters; and an interpolation processing module for inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence.

[0030] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein the processor is configured to implement the steps of an image processing method for a vehicle surround view system when executing a computer management program stored in the memory.

[0031] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium on which a computer management program is stored, and when the computer management program is executed by a processor, the steps of the image processing method of a vehicle surround view system are implemented.

[0032] The present invention provides an image processing method and system for a vehicle surround view system, which takes into account the influence of the camera's internal and external parameters and the vehicle's motion parameters on motion estimation and motion compensation, so that the generated intermediate frames are more realistic and a high-frame-rate panoramic video is obtained, which improves the user's viewing experience compared to traditional panoramic systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of an image processing method for a vehicle surround view system provided by the present invention;

[0034] Figure 2 A schematic diagram of a calculated pixel block motion vector;

[0035] Figure 3 A schematic diagram of the structure of an image processing system for a vehicle surround view system provided by the present invention;

[0036] Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0037] Figure 5 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0039] Embodiment 1

[0040] An image processing method for a vehicle surround view system, see Figure 1 , the image processing method comprises:

[0041] S1, receiving a low frame rate image sequence output by a vehicle surround view system camera.

[0042] It is understandable that when the video streams of the four cameras of the vehicle surround view system are input into the same chip, the video at this time is a low frame rate fisheye distorted video.

[0043] S2, calculate the motion vector of the same pixel block in two adjacent frame images.

[0044] As an embodiment, the calculation of the motion vector of the same pixel block on two adjacent frames of images includes: dividing each frame of image into multiple pixel blocks, searching for matching pixel blocks on two adjacent frames of images, the matching pixel blocks on two adjacent frames of images are the same pixel block on the two adjacent frames of images; obtaining the relative displacement between the matching pixel blocks on the two adjacent frames of images as the motion vector.

[0045] Specifically, for each image in the low frame rate image sequence, the image is divided into pixel blocks of a certain size, and the motion of the same pixel blocks between adjacent previous and next frames is compared to calculate the motion vector. The method is to use the search standard of minimizing the sum of the differences between pixel blocks, and use the three-dimensional recursive search method or the diamond method to traverse the previous and next frame images, and obtain the relative displacement of the current block (pixel block of the previous frame image) and the matching block (pixel frame of the next frame image) as the motion vector, which is expressed as Δx and Δy, which are the motion vectors of the block along the x and y axes respectively. See Figure 2 .

[0046] S3, correcting the motion vector according to the camera parameters and the vehicle motion parameters.

[0047] S4, based on the camera parameters, performs distortion correction on each of two adjacent frames of images.

[0048] As an embodiment, the correcting the motion vector according to the camera parameters and the self-vehicle motion parameters includes: calibrating the camera's intrinsic parameters and distortion parameters based on the Zhang Zhengyou calibration method, and obtaining the self-vehicle motion parameters; correcting the motion vector according to the camera's intrinsic parameters, distortion parameters and the self-vehicle motion parameters.

[0049] Due to the distortion of the surround camera image and the motion of the vehicle, the motion vector obtained in step S2 is inaccurate, so it is considered to use the camera's intrinsic parameters, extrinsic parameters and distortion parameters as well as the vehicle's motion parameters to correct the motion vector. The embodiment of the present invention uses the Zhang Zhengyou calibration method to obtain the camera's intrinsic parameters, extrinsic parameters and distortion parameters.

[0050] According to the Kannala-brandt distortion projection model, the conversion relationship from distorted image pixels to ideal image pixels can be obtained.

[0051] Its projection relationship is:

[0052]

[0053]

[0054] r 2 =x 2 +y 2 ;

[0055] θ==arctan(r);

[0056] θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 );

[0057] In the formula, (x', y') is the pixel coordinate of the image after fisheye distortion correction, (x, y) is the pixel coordinate of the fisheye distortion image output by the camera, and k1, k2, k3, k4 are the camera distortion parameters.

[0058] Considering the camera internal parameters, we can get the following relationship:

[0059] u=f x x′+C x ;

[0060] v=f y y′+C y ;

[0061] In the formula, f x 、f y , C x , C y is the camera internal parameter, (u,v) is the ideal image pixel coordinate.

[0062] According to the model, the corrected motion vector can be obtained:

[0063]

[0064]

[0065] In the formula, f x and f y is a constant, θ d and r are variables related to x and y.

[0066] Considering the longitudinal speed of the vehicle, the ideal motion vector can be obtained as follows:

[0067] Δu′=f(x,y,Δx)+αVt;

[0068] Δv′=f(x,y,Δy)+βVt;

[0069] In the formula, V is the longitudinal velocity of the vehicle, t is the time interval between two frames output by the camera, and α and β are calibrable coefficients.

[0070] S5, inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence.

[0071] Specifically, firstly, the two frames of images are subjected to distortion removal correction, as shown in steps S3 and S4.

[0072] Then, according to the unidirectional motion estimation method, N frames are inserted between the previous and next frames, and the inserted intermediate frames are numbered 1, 2, …, i, .., N. The pixel coordinates of the inserted frames can be obtained as:

[0073]

[0074] In the formula, (u, v) is the pixel coordinate of the previous frame after distortion correction in the previous and next frames, and Δu′ and Δv′ are the corrected motion vectors calculated according to step S3.

[0075] Through the above steps, the fisheye distortion video with low frame rate output by the camera is converted into an ideal image video with high frame rate.

[0076] As an embodiment, the low frame rate image sequence output by the camera of the vehicle surround view system is received, including: receiving the low frame rate image sequence output by multiple cameras of the vehicle surround view system; performing frame insertion processing on each low frame rate image sequence to obtain each high frame rate image sequence; and performing splicing and illumination uniformization processing on each high frame rate image sequence to obtain a high frame rate panoramic video.

[0077] Specifically, for the automobile surround view system, the high frame rate videos of the four cameras obtained in step S5 need to be post-processed, including perspective transformation, image stitching, and illumination homogenization processing, so as to finally obtain a smooth and realistic high frame rate panoramic video.

[0078] Perspective transformation can use an image-based homography matrix algorithm to obtain a bird's-eye view. Image stitching can be completed by establishing a unified coordinate system under a panoramic bird's-eye view through adjacent camera image reference points to complete image registration and stitching. Median filtering is performed on the seams of the stitched images to eliminate the seams and make the image smoother. Lighting homogenization processing can use the correction of the image RGB three-channel coefficients to minimize the color difference in the overlapping area of ​​the four images.

[0079] Embodiment 2

[0080] An image processing system for a vehicle surround view system, see Figure 3 The image processing system includes a receiving module 301, a calculating module 302, a correcting module 303 and an interpolation processing module 304.

[0081] The receiving module 301 is used to receive a low frame rate image sequence output by a vehicle surround view system camera; the calculating module 302 is used to calculate a motion vector of a same pixel block on two adjacent frames of images; the correcting module 303 is used to correct the motion vector according to camera parameters and vehicle motion parameters; and based on the camera parameters, perform distortion correction on each frame of two adjacent frames of images; the interpolation processing module 304 is used to insert an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence.

[0082] It can be understood that the image processing system of a vehicle surround view system provided by the present invention corresponds to the image processing method of the vehicle surround view system provided by the aforementioned embodiments. The relevant technical features of the image processing system of the vehicle surround view system can refer to the relevant technical features of the image processing method of the vehicle surround view system, which will not be repeated here.

[0083] Embodiment 3

[0084] See also Figure 4 , Figure 4 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides an electronic device 400, including a memory 410, a processor 420, and a computer program 411 stored in the memory 420 and executable on the processor 420. When the processor 420 executes the computer program 411, the following steps are implemented: receiving a low frame rate image sequence output by a vehicle surround view system camera; calculating a motion vector of a same pixel block on two adjacent frames of images; correcting the motion vector according to camera parameters and vehicle motion parameters; performing distortion correction on each of two adjacent frames of images based on the camera parameters; and inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence.

[0085] Embodiment 4

[0086] See also Figure 5 , Figure 5 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 5 As shown, this embodiment provides a computer-readable storage medium 500, on which a computer program 511 is stored. When the computer program 511 is executed by a processor, the following steps are implemented: receiving a low frame rate image sequence output by a vehicle surround view system camera; calculating a motion vector of a same pixel block on two adjacent frames of images; correcting the motion vector according to camera parameters and vehicle motion parameters; performing distortion correction on each of two adjacent frames of images based on the camera parameters; and inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence.

[0087] An image processing method and system for a vehicle surround view system provided by an embodiment of the present invention take into account the influence of the camera's internal and external parameters and the vehicle's motion parameters on motion estimation and motion compensation, so that the generated intermediate frames are more realistic and a high-frame-rate panoramic video is obtained, which improves the user's viewing experience compared to traditional panoramic systems.

[0088] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0093] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0094] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image processing method for a vehicle surround view system, characterized in that: include: Receiving a low frame rate image sequence output by a vehicle surround view system camera; Calculate the motion vector of the same pixel block in two adjacent frames of images; Correcting the motion vector according to camera parameters and ego-vehicle motion parameters; Based on the camera parameters, distortion correction is performed on each of two adjacent frames of images; According to the corrected motion vector, an intermediate frame image is inserted between two adjacent frames of images after distortion correction to obtain a high frame rate image sequence; The camera includes internal parameters and distortion parameters of the camera, and the motion vector is corrected according to the camera parameters and the vehicle motion parameters, including: Among them, (Δx, Δy) is the motion vector of the pixel block along the x and y axes, is the motion parameter of the vehicle, and f x 、f y is the camera internal parameter, θ d , r is determined according to the camera distortion parameters and the coordinates of the pixel block, V is the longitudinal velocity of the vehicle, t is the time interval between two adjacent frames, and α and β are calibrable coefficients.

2. The image processing method according to claim 1, characterized in that: The calculating of the motion vector of the same pixel block in two adjacent frames of images includes: Divide each frame of image into a plurality of pixel blocks, and search for matching pixel blocks on two adjacent frame images, wherein the matching pixel blocks on two adjacent frame images are the same pixel blocks on the two adjacent frame images; The relative displacement between the matching pixel blocks on two adjacent frames of images is obtained as a motion vector.

3. The image processing method according to claim 1 or 2, characterized in that: The correcting the motion vector according to the camera parameters and the vehicle motion parameters includes: Calibrate the camera's intrinsic parameters and distortion parameters based on Zhang Zhengyou's calibration method, and obtain the vehicle's motion parameters; The motion vector is corrected according to the camera's intrinsic parameters, distortion parameters and the vehicle's motion parameters.

4. The image processing method according to claim 1, characterized in that: The method of performing distortion correction on each of two adjacent frames of images based on the camera parameters includes: According to the Kannala-brandt distortion projection model, the conversion relationship between the image pixels before distortion correction and the image pixels after distortion correction is established: r 2 =x 2 +y 2 ; θ = arctan (r); i d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 ); Where (x', y') is the pixel coordinate of the image after fisheye distortion correction, (x, y) is the pixel coordinate of the distorted image output by the camera, and k1, k2, k3, k4 are the camera distortion parameters; Based on the camera intrinsic parameters, the pixel coordinates of the image after distortion correction are converted to the ideal image pixel coordinates: u=f x x ′ +C x ; v=f y y ′ +C y ; In the formula, f x 、f y , C x , C y is the camera internal parameter, (u,v) is the ideal image pixel coordinate.

5. The image processing method according to claim 4, characterized in that: The method inserting an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence includes: According to the unidirectional motion estimation method, N frames of images are inserted between two adjacent frames of images, and the inserted intermediate frames are numbered 1, 2, ..., i, .., N; Among them, the pixel coordinates of each inserted intermediate frame image are: Where (u, v) is the pixel coordinate of the previous frame of the previous two frames, Δu ′ and Δv ′ is the corrected motion vector; A plurality of intermediate frame images are inserted between every two frames of images in the low frame rate image sequence to obtain a high frame rate image sequence.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The receiving of a low frame rate image sequence output by a vehicle surround view system camera includes: Receiving low frame rate image sequences output by multiple cameras of a vehicle surround view system; Perform frame insertion processing on each low frame rate image sequence to obtain each high frame rate image sequence; Each high frame rate image sequence is stitched and the illumination is evenly processed to obtain a high frame rate panoramic video.

7. An image processing system for a vehicle surround view system, characterized in that: include: A receiving module, used for receiving a low frame rate image sequence output by a vehicle surround view system camera; A calculation module, used for calculating the motion vector of the same pixel block in two adjacent frames of images; A correction module, used to correct the motion vector according to the camera parameters and the vehicle motion parameters; and to perform distortion correction on each of two adjacent frames of images based on the camera parameters; An interpolation processing module is used to insert an intermediate frame image between two adjacent frames of images after distortion correction according to the corrected motion vector to obtain a high frame rate image sequence; The camera includes internal parameters and distortion parameters of the camera, and the motion vector is corrected according to the camera parameters and the vehicle motion parameters, including: Among them, (Δx, Δy) is the motion vector of the pixel block along the x and y axes, is the motion parameter of the vehicle, and f x 、f y is the camera internal parameter, θ d , r is determined according to the camera distortion parameters and the coordinates of the pixel block, V is the longitudinal velocity of the vehicle, t is the time interval between two adjacent frames, and α and β are calibrable coefficients.

Images