Image processing method and device

Abstract

The embodiment of the invention discloses an image processing method and device. The method comprises the steps that scene images collected by a plurality of cameras respectively are acquired; the plurality of cameras are respectively deployed at different positions of a target vehicle body and are used for collecting scene images in different directions around the target vehicle; converting each scene image into corresponding texture data, and storing the texture data in a video memory; determining a corresponding relationship between a pixel position in a target top view image to be generated and each pixel position in the scene image for each scene image through a shader program running on the GPU, and converting texture data corresponding to the scene image in the video memory into target color data to obtain a sub top view image corresponding to the scene image; and combining the sub top view images corresponding to the scene images through a shader program to obtain a target top view image. According to the method, the workload of a CPU (Central Processing Unit) during top view generation can be reduced, and smooth operation in an embedded platform with limited computing power can be realized.

Description

Technical Field

[0001] The present application relates to the field of assisted driving technology, and in particular to an image processing method and device. Background Art

[0002] Automobile assisted driving technology is conducive to improving the safety and convenience of users' driving. Image-based assisted driving technology is one of the most common assisted driving technologies. The general reversing image system installs a single rear-view camera at the rear of the vehicle. The rear-view camera transmits the image it collects back and displays it on the corresponding device at the driver's seat.

[0003] However, the above solution can only let the driver know the scene at the rear of the car, and cannot eliminate the blind spots of the A-pillar, B-pillar and hood. In order to solve this problem, the relevant technology installs a camera around the vehicle, and generates a bird's-eye view by image stitching to help the driver understand the situation around the vehicle more comprehensively and accurately, so as to obtain a safer and higher-quality driving experience. However, most of the current open source bird's-eye view generation algorithms are based on the central processing unit (CPU), which generally generates a lot of computing consumption and is not conducive to use in embedded platforms such as car computers. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and device, which can reduce the workload of the CPU when generating a top view, and ensure that the top view generation algorithm runs smoothly on an embedded platform with limited computing power.

[0005] In view of this, the first aspect of the present application provides an image processing method, the method comprising:

[0006] Acquire scene images captured by multiple cameras respectively; the multiple cameras are respectively deployed at different positions of the target vehicle body, and are used to capture scene images at different directions around the target vehicle;

[0007] Convert each of the scene images into corresponding texture data, and save the texture data into a video memory;

[0008] By means of a shader program running on a graphics processor, for each of the scene images, a correspondence between a pixel position in a target bird's-eye view image to be generated and each pixel position in the scene image is determined, and texture data corresponding to the scene image in the video memory is converted into target color data to obtain a sub-bird's-eye view image corresponding to the scene image;

[0009] The sub-top-view images corresponding to the respective scene images are combined through the shader program to obtain the target top-view image.

[0010] Optionally, the scene image is in yuv format, and the total size of the image data of the scene image is bytes, of which the first h×w bytes are the brightness channel data of the scene image, and the latter The bytes of data are the chrominance channel data of the scene image;

[0011] The converting each of the scene images into corresponding texture data comprises:

[0012] For each of the scene images, the brightness channel data of the scene image is read as single-channel texture data of size h×w and transferred to the video memory; the chrominance channel data of the scene image is read as single-channel texture data of size The dual-channel texture data is transferred to the video memory.

[0013] Optionally, determining the correspondence between the pixel position in the target overhead image to be generated and each pixel position in the scene image includes:

[0014] Determine a first correspondence between a pixel position in the target overhead image and each pixel position in a dedistorted image corresponding to the scene image;

[0015] A second correspondence between a pixel position in the dedistorted image and each pixel position in the scene image is determined.

[0016] Optionally, determining a first correspondence between a pixel position in the target overhead image and each pixel position in a dedistorted image corresponding to the scene image includes:

[0017] The first corresponding relationship between the pixel coordinates in the target overhead image and each pixel coordinate in the dedistorted image is determined by the following equations (1) and (2):

[0018] [x′, y′, z′] = H -1 [u, v, 1] (1)

[0019]

[0020] Among them, H is the homography matrix of the corresponding direction; u and v are the pixel coordinates in the target overhead image; u′ and v′ are the pixel coordinates in the dedistorted image; x′, y′ and z′ are the intermediate parameters in the pixel coordinate conversion process.

[0021] Optionally, determining a second corresponding relationship between a pixel position in the dedistorted image and each pixel position in the scene image includes:

[0022] The second corresponding relationship between the pixel coordinates in the dedistorted image and each pixel coordinate in the scene image is determined by the following equations (3) to (9):

[0023] [x2, y2, z2] = K -1 [u′, v′, 1] (3)

[0024]

[0025]

[0026] θ=arctan(r / f) (6)

[0027] θ d =θ(1+d1θ 2 +d2θ 4 +d3θ 6 +d4θ 8 ) (7)

[0028] x1=(θ d / r)a,y1=(θ d / r)b (8)

[0029] [u″, v″, 1]=K·[x1, y1, 1] (9)

[0030] Wherein, K corresponds to the camera matrix in the direction; u′ and v′ are the pixel coordinates in the dedistorted image; x2, y2 and z2 are the unnormalized camera coordinates; a and b are the normalized camera coordinates; r is the normalized incident radius; θ is the incident angle after dedistortion processing; f is the focal length of the camera used to capture the scene image, which defaults to 1; θ d is the incident angle without dedistortion processing, d1, d2, d3 and d4 are distortion parameters respectively; x1 and y1 are the normalized camera coordinates corresponding to the scene image; u″ and v″ are the pixel coordinates of the scene image.

[0031] Optionally, the texture data includes y channel data, u channel data, and v channel data corresponding to pixels in the scene image; and converting the texture data corresponding to the scene image in the video memory into target color data includes:

[0032] The y channel data, u channel data and v channel data corresponding to the pixel point are converted into the r channel data, g channel data and b channel data corresponding to the pixel point by using the following equations (10) to (13):

[0033] y_t=m1*(y-m2) (10)

[0034] r=y_t+m3*v (11)

[0035] g=y_t-m4*u-m5*v (12)

[0036] b=y_t+m6*u (13)

[0037] Among them, y, u and v respectively represent the y channel data, u channel data and v channel data corresponding to the pixel point, and y_t is an intermediate parameter in the conversion process; r, g and b respectively represent the r channel data, g channel data and b channel data corresponding to the pixel point; m1, m2, m3, m4, m5 and m6 are all transformation parameters used when converting y channel data, u channel data and v channel data into r channel data, g channel data and b channel data.

[0038] Optionally, for each of the scene images, determining a correspondence between a pixel position in a target overhead view image to be generated and each pixel position in the scene image, and converting texture data corresponding to the scene image in the video memory into target color data to obtain a sub-overhead view image corresponding to the scene image, includes:

[0039] Determining an initial sub-top view image corresponding to the scene image based on a correspondence between a pixel position in the target top view image and each pixel position in the scene image, and the target color data;

[0040] The invalid portion of the initial sub-top-view image is removed by using the image mask corresponding to the scene image to obtain the sub-top-view image corresponding to the scene image.

[0041] Optionally, combining the sub-top-view images corresponding to the respective scene images through the shader program to obtain the target top-view image includes:

[0042] The texture data of the sub-top-view images corresponding to each of the scene images are added together through the shader program to obtain the target top-view image.

[0043] Optionally, the method further includes:

[0044] Draw reversing auxiliary lines and virtual radar walls through the target drawing tool;

[0045] The reversing auxiliary line and the virtual radar wall are covered in the target top view image.

[0046] A second aspect of the present application provides an image processing device, the device comprising:

[0047] An image acquisition module, used to acquire scene images captured by multiple cameras; the multiple cameras are respectively deployed at different positions of the target vehicle body, and are used to capture scene images at different positions around the target vehicle;

[0048] A texture conversion module, used for converting each of the scene images into corresponding texture data, and saving the texture data into a video memory;

[0049] A top-view conversion module is used to determine, for each of the scene images, a correspondence between a pixel position in a target top-view image to be generated and each pixel position in the scene image through a shader program running on a graphics processor, and to convert texture data corresponding to the scene image in the video memory into target color data to obtain a sub-top-view image corresponding to the scene image;

[0050] The bird's-eye view generation module is used to combine the sub-bird's-eye view images corresponding to each of the scene images through the shader program to obtain the target bird's-eye view image.

[0051] Optionally, the scene image is in yuv format, and the total size of the image data of the scene image is bytes, of which the first h×w bytes are the brightness channel data of the scene image, and the latter Bytes of data are the chromaticity channel data of the scene image; then the texture conversion module is specifically used for:

[0052] For each of the scene images, the brightness channel data of the scene image is read as single-channel texture data of size h×w and transferred to the video memory; the chrominance channel data of the scene image is read as single-channel texture data of size The dual-channel texture data is transferred to the video memory.

[0053] Optionally, the top view conversion module is specifically used for:

[0054] Determine a first correspondence between a pixel position in the target overhead image and each pixel position in a dedistorted image corresponding to the scene image;

[0055] A second correspondence between a pixel position in the dedistorted image and each pixel position in the scene image is determined.

[0056] Optionally, the top view conversion module is specifically used for:

[0057] The first corresponding relationship between the pixel coordinates in the target overhead image and each pixel coordinate in the dedistorted image is determined by the following equations (1) and (2):

[0058] [x′, y′, z′] = H -1 [u, v, 1] (1)

[0059]

[0060] Among them, H is the homography matrix of the corresponding direction; u and v are the pixel coordinates in the target overhead image; u′ and v′ are the pixel coordinates in the dedistorted image; x′, y′ and z′ are the intermediate parameters in the pixel coordinate conversion process.

[0061] Optionally, the top view conversion module is specifically used for:

[0062] The second corresponding relationship between the pixel coordinates in the dedistorted image and each pixel coordinate in the scene image is determined by the following equations (3) to (9):

[0063] [x2, y2, z2] = K -1 [u′, v′, 1] (3)

[0064]

[0065]

[0066] θ=arctan(r / f) (6)

[0067] θ d =θ(1+d1θ 2 +d2θ 4 +d3θ 6 +d4θ 8 ) (7)

[0068] x1=(θ d / r)a,y1=(θ d / r)b (8)

[0069] [u″,v″,1]=K·[x1,y1,1] (9)

[0070] Wherein, K corresponds to the camera matrix in the direction; u′ and v′ are the pixel coordinates in the dedistorted image; x2, y2 and z2 are the unnormalized camera coordinates; a and b are the normalized camera coordinates; r is the normalized incident radius; θ is the incident angle after dedistortion processing; f is the focal length of the camera used to capture the scene image, which defaults to 1; θ d is the incident angle without dedistortion processing, d1, d2, d3 and d4 are distortion parameters respectively; x1 and y1 are the normalized camera coordinates corresponding to the scene image; u″ and v″ are the pixel coordinates of the scene image.

[0071] Optionally, the texture data includes y channel data, u channel data and v channel data corresponding to the pixel points in the scene image; then the top view conversion module is specifically used for:

[0072] The y channel data, u channel data and v channel data corresponding to the pixel point are converted into the r channel data, g channel data and b channel data corresponding to the pixel point by using the following equations (10) to (13):

[0073] y_t=m1*(y-m2) (10)

[0074] r=y_t+m3*v (11)

[0075] g=y_t-m4*u-m5*v (12)

[0076] b=y_t+m6*u (13)

[0077] Among them, y, u and v respectively represent the y channel data, u channel data and v channel data corresponding to the pixel point, and y_t is an intermediate parameter in the conversion process; r, g and b respectively represent the r channel data, g channel data and b channel data corresponding to the pixel point; m1, m2, m3, m4, m5 and m6 are all transformation parameters used when converting y channel data, u channel data and v channel data into r channel data, g channel data and b channel data.

[0078] Optionally, the top view conversion module is specifically used for:

[0079] Determining an initial sub-top view image corresponding to the scene image based on a correspondence between a pixel position in the target top view image and each pixel position in the scene image, and the target color data;

[0080] The invalid portion of the initial sub-top-view image is removed by using the image mask corresponding to the scene image to obtain the sub-top-view image corresponding to the scene image.

[0081] Optionally, the top view generating module is specifically used for:

[0082] The texture data of the sub-top-view images corresponding to each of the scene images are added together through the shader program to obtain the target top-view image.

[0083] Optionally, the device further comprises:

[0084] The auxiliary information drawing module is used to draw a reversing auxiliary line and a virtual radar wall through a target drawing tool; and cover the reversing auxiliary line and the virtual radar wall in the target top view image.

[0085] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0086] An embodiment of the present application provides an image processing method, which includes: acquiring scene images captured by multiple cameras, where the multiple cameras are respectively deployed at different positions on the body of a target vehicle, and are used to capture scene images at different directions around the target vehicle; then, converting each scene image into corresponding texture data, and saving the converted texture data into a video memory; then, through a shader program running on a graphics processing unit (GPU), for each scene image, determining the correspondence between the pixel position in a target overhead image to be generated and each pixel position in the scene image, and converting the texture data corresponding to the scene image in the video memory into target color data to obtain a sub-overhead image corresponding to the scene image; finally, through the shader program, combining the sub-overhead images corresponding to each of the scene images to obtain the target overhead image. In this way, by using a GPU with strong parallel computing capabilities to perform overhead transformation, image data conversion and other processing operations in the overhead view generation process, multiple pixel points in the scene image captured by the camera can be transformed into pixel points in the target overhead view image at the same time and in parallel. This can greatly improve the calculation speed, reduce the delay in overhead view image generation, and improve the quality of the generated overhead view image. Accordingly, the CPU workload is reduced, ensuring that the overhead view generation algorithm runs smoothly on an embedded platform with limited computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0088] Figure 2 A flowchart of another image processing method provided in an embodiment of the present application;

[0089] Figure 3 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0091] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0092] It should be noted that GPU is a high-performance parallel computing device, whose main function is to accelerate graphics computing, computer vision and other tasks; GPU has stronger parallel computing capabilities than CPU, and can process and calculate large amounts of data in parallel, thereby improving the computing speed; in the embodiment of the present application, GPU acceleration is used to generate overhead images based on scene images collected by various cameras on the vehicle body, so as to improve the efficiency of generating overhead images.

[0093] Open Graphics Library (OpenGL) is a cross-platform graphics application programming interface (Application Programming Interface, API), whose main function is to provide a standard way for programmers to write cross-platform graphics applications; OpenGL can use GPU to achieve efficient graphics calculation and rendering. Shader program is a program that runs on GPU, and its main function is to control the color, lighting and shadow effects of pixels.

[0094] By writing shader programs through OpenGL, operations such as image perspective transformation and image format conversion can be run on the GPU, thereby greatly improving the operation speed, reducing the delay in the generation of overhead images, and improving the picture quality of the generated overhead images. In the embodiment of the present application, the fisheye image dedistortion, overhead transformation and image data conversion processes are executed in parallel through shader programs running on the GPU, thereby minimizing the CPU workload, so that the overhead image generation process can run smoothly on an embedded platform with limited computing power.

[0095] See also Figure 1 , Figure 1A flowchart of an image processing method provided in an embodiment of the present application. It should be noted that the method is applicable to a two-dimensional display system including multiple cameras; illustratively, a camera can be installed on the front, back, left, and right sides of the target vehicle to capture the scene around the vehicle body, and then, based on the images captured by each camera, a top-view image with the geometric center of the target vehicle as the origin is synthesized, and the image is rendered on the vehicle screen.

[0096] In addition, before executing the image processing method, the camera matrix and homography matrix required for image transformation can be initialized, and image masks corresponding to each camera can be generated. The image masks are stored in the video memory as single-channel textures, and the shader program on the GPU can be initialized at the same time.

[0097] like Figure 1 As shown, the image processing method provided in the embodiment of the present application includes the following steps:

[0098] Step 101: Acquire scene images captured by multiple cameras respectively; the multiple cameras are respectively deployed at different positions of the target vehicle body to capture scene images at different directions around the target vehicle.

[0099] In an embodiment of the present application, scene images captured by multiple cameras installed at different positions on the target vehicle body at the same time can be obtained; for example, assuming that a camera is installed on the front, back, left and right sides of the target vehicle body, then the scene images captured by these four cameras can be obtained at this time; it should be understood that in actual applications, cameras can also be installed at other positions on the target vehicle body, and the number of installed cameras can be more or less, and the embodiment of the present application does not impose any limitation on this.

[0100] It should be noted that the scene image captured by the camera here can specifically be a fisheye image. After the fisheye images captured by each camera are obtained, they can be saved in the memory in yuv420 format.

[0101] Step 102: Convert each of the scene images into corresponding texture data, and save the texture data into a video memory.

[0102] Then, each acquired scene image can be converted into corresponding texture data, and the converted texture data can be stored in the video memory controlled by the GPU. For scene images in yuv420 format, the y channel data and uv channel data of the scene image can be converted into GPU textures respectively and stored in the video memory.

[0103] In a possible implementation, when the scene image is in yuv format (such as yuv420 format), the total size of the image data of the scene image is bytes, of which the first h×w bytes are the brightness channel data of the scene image, and the latter The byte data is the chrominance channel data of the scene image. At this point, the conversion and storage of texture data can be achieved through the following steps:

[0104] Step 1021: For each scene image, read the brightness channel data of the scene image, that is, read the first h×w bytes of data in the scene image as single-channel texture data of size h×w, and transfer the single-channel texture data of size h×w to the video memory.

[0105] Step 1022: Read the chromaticity channel data of the scene image, that is, read the chromaticity channel data of the scene image. bytes of data, as the size The dual-channel texture data is converted into The dual-channel texture data is transferred to the video memory.

[0106] Step 103: Through the shader program running on the graphics processor, for each of the scene images, the correspondence between the pixel position in the target overhead view image to be generated and each pixel position in the scene image is determined, and the texture data corresponding to the scene image in the video memory is converted into target color data to obtain a sub-overhead view image corresponding to the scene image.

[0107] Then, based on the initialized camera matrix, fisheye camera distortion parameters and homography matrix, pre-generated image mask, and texture data generated by the above step 102, according to the fisheye de-distortion and bird's-eye view transformation principle, the corresponding relationship between the pixel points in the target bird's-eye view image to be generated and the pixel points in the scene image originally collected by the camera is determined through the shader program running on the GPU, and the image data is converted from the YUV format to the RGB (red green blue) format, thereby realizing the transformation of the pixel points in the original scene image collected by the camera to the pixel points in the target bird's-eye view image, and obtaining the sub-bird's-eye view image corresponding to the scene image. Since the shader program has a high parallel computing capability, it can realize the transformation process of multiple pixel points at the same time, so the computing efficiency can be improved.

[0108] In a possible implementation, the following steps may be performed to generate a corresponding sub-top view image for each scene image:

[0109] Step 1031: Determine a first correspondence between a pixel position in the target overhead image and each pixel position in the dedistorted image corresponding to the scene image.

[0110] Specifically, the first correspondence between the pixel coordinates in the target overhead image and each pixel coordinate in the dedistorted image corresponding to the scene image can be determined in the GPU by using the following equations (1) and (2):

[0111] [x′,y′,z′]=H -1 ·[u,v,1](1)

[0112]

[0113] Among them, H is the homography matrix of the corresponding direction; u and v are the pixel coordinates in the target top-view image; u ′ and v′ are the pixel coordinates in the dedistorted image; x′, y′, and z′ are the intermediate parameters in the pixel coordinate conversion process.

[0114] Step 1032: Determine a second correspondence between a pixel position in the dedistorted image and each pixel image in the scene image.

[0115] Specifically, the second corresponding relationship between the pixel coordinates in the dedistorted image and each pixel coordinate in the original scene image can be determined in the GPU by using the following equations (3) to (9):

[0116] [x2, y2, z2] = K -1 [u′, v′, 1] (3)

[0117]

[0118]

[0119] θ=arctan(r / f)(f=1) (6)

[0120] θ d =θ(1+d1θ 2 +d2θ 4 +d3θ 6 +d4θ 8 ) (7)

[0121] x1=(θ d / r)a,y1=(θ d / r)b (8)

[0122] [u″, v″, 1]=K·[x1, y1, 1] (9)

[0123] Where K is the camera matrix of the corresponding direction; u′ and v′ are the pixel coordinates in the dedistorted image; x2, y2, and z2 are the unnormalized camera coordinates; a and b are the normalized camera coordinates; r is the incident radius; θ is the incident angle after dedistortion processing; f is the focal length of the camera used to capture the scene image, which is set to 1 in this step; θ d is the incident angle without dedistortion processing, d1, d2, d3 and d4 are distortion parameters respectively; x1 and y1 are the normalized camera coordinates corresponding to the scene image; u″ and v″ are the pixel coordinates of the scene image.

[0124] Step 1033: Based on the correspondence between the pixel positions in the target overhead image and the pixel positions in the scene image, the y channel data, u channel data and v channel data corresponding to the pixel points in the scene image are obtained from the texture data stored in the video memory.

[0125] Step 1034: convert the y channel data, u channel data and v channel data corresponding to the pixel points in the scene image into corresponding r channel data, g channel data and b channel data.

[0126] Specifically, the y channel data, u channel data, and v channel data corresponding to the pixel point can be converted into the r channel data, g channel data, and b channel data corresponding to the pixel point in the GPU through the following equations (10) to (13):

[0127] y_t=m1*(y-m2) (10)

[0128] r=y_t+m3*v (11)

[0129] g=y_t-m4*u-m5*v (12)

[0130] b=y_t+m6*u(13)

[0131] Among them, y, u and v represent the y channel data, u channel data and v channel data corresponding to the pixel point respectively, and y_t is an intermediate parameter in the conversion process; r, g and b represent the r channel data, g channel data and b channel data corresponding to the pixel point respectively; m1, m2, m3, m4, m5 and m6 are all transformation parameters used when converting y channel data, u channel data and v channel data into r channel data, g channel data and b channel data. For example, m1 can be equal to 1.16438356, m2 can be equal to 0.0625, m3 can be equal to 1.59602678, m4 can be equal to 0.39176229, m5 can be equal to 0.81296764, and m6 can be equal to 2.01723214.

[0132] In this way, through the above process, based on the correspondence between the pixel points in the target overhead image and the pixel points in the scene image, the mapping position of the pixel points in the scene image in the target overhead image is determined, and the YUV data of the pixel points in the scene image is converted into corresponding RGB data, so as to obtain the initial sub-overhead image corresponding to the scene image.

[0133] Step 1035: using the image mask corresponding to the scene image, remove the invalid part of the initial sub-top-view image to obtain the sub-top-view image corresponding to the scene image.

[0134] That is, the initial sub-top view image obtained by the above steps is multiplied by the value of the corresponding position of the image mask to remove the invalid part of the initial sub-top view image, and obtain the sub-top view image corresponding to the scene image. The image mask here has a corresponding relationship with the camera that captures the scene image, and includes several values ​​between 0 and 1.

[0135] Step 104: combining the sub-top-view images corresponding to the respective scene images through the shader program to obtain the target top-view image.

[0136] Furthermore, the sub-top-view images corresponding to each scene image are combined through the shader program, that is, the textures corresponding to the multiple scene images generated through the above steps are passed into the shader program, so that the textures in multiple directions are directly added through the shader program to obtain a complete target top-view image, and the target top-view image is rendered on the car screen.

[0137] Optionally, reversing auxiliary lines and virtual radar walls can also be drawn through target drawing tools, such as drawing reversing auxiliary lines and radar walls through drawing tool libraries such as cairo or Skia, or directly drawing through opengl; the drawn reversing auxiliary lines and virtual radar walls are overlaid on the target overhead image, thereby completing the display of the entire assisted driving function.

[0138] The image processing method provided in the embodiment of the present application uses a GPU with strong parallel computing capabilities to perform overhead transformation, image data conversion and other processing operations in the overhead view generation process, and can simultaneously and parallelly transform multiple pixel points in the scene image captured by the camera into pixel points in the target overhead view image. This can greatly improve the calculation speed, reduce the delay in the generation of the overhead view image, and at the same time improve the quality of the generated overhead view image; accordingly, the CPU workload is reduced, ensuring that the overhead view generation algorithm runs smoothly on an embedded platform with limited computing power.

[0139] See also Figure 2 , Figure 2Another flowchart of the image processing method provided in the embodiment of the present application. This embodiment takes the installation of a camera at the front and rear of the target vehicle body to capture the scene around the target vehicle body as an example. After the four cameras are installed, the cameras can be calibrated at the calibration site, that is, the homography matrix corresponding to the camera is calculated according to the image captured by the camera in combination with the grid arranged in the calibration site, and saved in the system.

[0140] In specific implementation, the parameters and the camera may be initialized first, that is, the pre-generated calibration information and texture information are read and the camera is started, and at the same time, the data stored in the buffer is cleared, that is, the result of the last rendering and the original data in the video memory are cleared.

[0141] Then, try to read the image captured by the camera, that is, read the current frame image captured by a single camera. If the reading fails, temporarily skip the camera, that is, consider the camera data error, temporarily abandon the image generation in this direction, and continue to read the current frame image captured by other cameras. If the reading is successful, convert the read image into a texture, that is, transfer the read yuv420 format image to the video memory through OpenGL through the above step 102; then, calculate the sub-top view image corresponding to the image, that is, the top view image in a single direction, that is, through the shader program of the GPU, use the above step 103 to calculate the top view image in a single direction.

[0142] If after the calculation of the overhead image in a certain direction is completed, there are still overhead images in other directions that have not been calculated, then try to read the images collected by the cameras in other directions, and determine the overhead images in that direction accordingly. If the calculation of the overhead images in all directions has been completed, then through the shader program of the GPU, the overhead images in all directions are combined using the above step 104 to obtain the final target overhead image that completely covers the scene information in all directions. And through 2D drawing tools such as cairo or skia, draw reversing auxiliary lines, radar walls and other information on the target overhead image.

[0143] The present application also provides an image processing device. Figure 3 , Figure 3 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the device comprises:

[0144] An image acquisition module 301 is used to acquire scene images acquired by multiple cameras; the multiple cameras are respectively deployed at different positions of the target vehicle body, and are used to acquire scene images at different positions around the target vehicle;

[0145] The texture conversion module 302 is used to convert each of the scene images into corresponding texture data and save the texture data into a video memory;

[0146] The top view conversion module 303 is used to determine, for each of the scene images, the correspondence between the pixel position in the target top view image to be generated and each pixel position in the scene image through the shader program running on the graphics processor, and convert the texture data corresponding to the scene image in the video memory into target color data to obtain a sub-top view image corresponding to the scene image;

[0147] The bird's-eye view generation module 304 is used to combine the sub-bird's-eye view images corresponding to the respective scene images through the shader program to obtain the target bird's-eye view image.

[0148] Optionally, the scene image is in yuv format, and the total size of the image data of the scene image is bytes, of which the first h×w bytes are the brightness channel data of the scene image, and the latter Bytes of data are the chromaticity channel data of the scene image; then the texture conversion module 302 is specifically used for:

[0149] For each of the scene images, the brightness channel data of the scene image is read as single-channel texture data of size h×w and transferred to the video memory; the chrominance channel data of the scene image is read as single-channel texture data of size The dual-channel texture data is transferred to the video memory.

[0150] Optionally, the top view conversion module 303 is specifically used for:

[0151] Determine a first correspondence between a pixel position in the target overhead image and each pixel position in a dedistorted image corresponding to the scene image;

[0152] A second correspondence between a pixel position in the dedistorted image and each pixel position in the scene image is determined.

[0153] Optionally, the top view conversion module 303 is specifically used for:

[0154] The first corresponding relationship between the pixel coordinates in the target overhead image and each pixel coordinate in the dedistorted image is determined by the following equations (1) and (2):

[0155] [x′,y′,z′]=H -1 ·[u,v,1] (1)

[0156]

[0157] Among them, H is the homography matrix of the corresponding direction; u and v are the pixel coordinates in the target overhead image; u′ and v′ are the pixel coordinates in the dedistorted image; x′, y′ and z′ are the intermediate parameters in the pixel coordinate conversion process.

[0158] Optionally, the top view conversion module 303 is specifically used for:

[0159] The second corresponding relationship between the pixel coordinates in the dedistorted image and each pixel coordinate in the scene image is determined by the following equations (3) to (9):

[0160] [x2,y2,z2]=K -1 ·[u′,v′,1] (3)

[0161]

[0162]

[0163] θ=arctan(r / f) (6)

[0164] θ d =θ(1+d1θ 2 +d2θ 4 +d3θ 6 +d4θ 8 ) (7)

[0165] x1=(θ d / r)a,y1=(θ d / r)b (8)

[0166] [u″,v″,1]=K·[x1,y1,1] (9)

[0167] Wherein, K corresponds to the camera matrix in the direction; u′ and v′ are the pixel coordinates in the dedistorted image; x2, y2 and z2 are the unnormalized camera coordinates; a and b are the normalized camera coordinates; r is the normalized incident radius; θ is the incident angle after dedistortion processing; f is the focal length of the camera used to capture the scene image, which defaults to 1; θ d is the incident angle without dedistortion processing, d1, d2, d3 and d4 are distortion parameters respectively; x1 and y1 are the normalized camera coordinates corresponding to the scene image; u″ and v″ are the pixel coordinates of the scene image.

[0168] Optionally, the texture data includes y channel data, u channel data and v channel data corresponding to the pixel points in the scene image; then the top view conversion module 303 is specifically used for:

[0169] The y channel data, u channel data and v channel data corresponding to the pixel point are converted into the r channel data, g channel data and b channel data corresponding to the pixel point by using the following equations (10) to (13):

[0170] y_t=m1*(y-m2) (10)

[0171] r=y_t+m3*v (11)

[0172] g=y_t-–-m4*u-–m5*v (12)

[0173] b=y_t+m6*u (13)

[0174] Among them, y, u and v respectively represent the y channel data, u channel data and v channel data corresponding to the pixel point, and y_t is an intermediate parameter in the conversion process; r, g and b respectively represent the r channel data, g channel data and b channel data corresponding to the pixel point; m1, m2, m3, m4, m5 and m6 are all transformation parameters used when converting y channel data, u channel data and v channel data into r channel data, g channel data and b channel data.

[0175] Optionally, the top view conversion module 303 is specifically used for:

[0176] Determining an initial sub-top view image corresponding to the scene image based on a correspondence between a pixel position in the target top view image and each pixel position in the scene image, and the target color data;

[0177] The invalid portion of the initial sub-top-view image is removed by using the image mask corresponding to the scene image to obtain the sub-top-view image corresponding to the scene image.

[0178] Optionally, the top view generating module 304 is specifically used for:

[0179] The texture data of the sub-top-view images corresponding to each of the scene images are added together through the shader program to obtain the target top-view image.

[0180] Optionally, the device further comprises:

[0181] The auxiliary information drawing module is used to draw a reversing auxiliary line and a virtual radar wall through a target drawing tool; and cover the reversing auxiliary line and the virtual radar wall in the target top view image.

[0182] The image processing device provided in the embodiment of the present application uses a GPU with strong parallel computing capabilities to perform overhead transformation, image data conversion and other processing operations in the overhead view generation process, and can simultaneously and parallelly transform multiple pixel points in the scene image captured by the camera into pixel points in the target overhead view image. This can greatly improve the calculation speed, reduce the delay in the generation of the overhead view image, and at the same time improve the quality of the generated overhead view image; accordingly, the CPU workload is reduced, ensuring that the overhead view generation algorithm runs smoothly on an embedded platform with limited computing power.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store computer programs.

[0188] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0189] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire scene images captured by multiple cameras respectively; the multiple cameras are respectively deployed at different positions of the target vehicle body, and are used to capture scene images at different directions around the target vehicle; Convert each of the scene images into corresponding texture data, and save the texture data into a video memory; By means of a shader program running on a graphics processor, for each of the scene images, a correspondence between a pixel position in a target bird's-eye view image to be generated and each pixel position in the scene image is determined, and texture data corresponding to the scene image in the video memory is converted into target color data to obtain a sub-bird's-eye view image corresponding to the scene image; The sub-top-view images corresponding to the respective scene images are combined through the shader program to obtain the target top-view image.

2. The method according to claim 1, characterized in that: The scene image is in yuv format, and the total size of the image data of the scene image is bytes, of which the first h×w bytes are the brightness channel data of the scene image, and the latter The bytes of data are the chrominance channel data of the scene image; The converting each of the scene images into corresponding texture data comprises: For each of the scene images, the brightness channel data of the scene image is read as single-channel texture data of size h×w and transferred to the video memory; the chrominance channel data of the scene image is read as single-channel texture data of size The dual-channel texture data is transferred to the video memory.

3. The method according to claim 1, characterized in that The determining of the correspondence between the pixel position in the target overhead image to be generated and each pixel position in the scene image includes: Determine a first correspondence between a pixel position in the target overhead image and each pixel position in a dedistorted image corresponding to the scene image; A second correspondence between a pixel position in the dedistorted image and each pixel position in the scene image is determined.

4. The method according to claim 3, characterized in that: The determining a first corresponding relationship between a pixel position in the target overhead image and each pixel position in the dedistorted image corresponding to the scene image comprises: The first corresponding relationship between the pixel coordinates in the target overhead image and each pixel coordinate in the dedistorted image is determined by the following equations (1) and (2): [x′,y′,z′]=H -1 ·[u,v,1](1) Where H is the homography matrix of the corresponding direction; u and v are the pixel coordinates in the target top view image; u ′ and v′ are pixel coordinates in the dedistorted image; x′, y′ and z′ are intermediate parameters in the pixel coordinate conversion process.

5. The method according to claim 3, characterized in that: The determining a second corresponding relationship between a pixel position in the dedistorted image and each pixel position in the scene image comprises: The second corresponding relationship between the pixel coordinates in the dedistorted image and each pixel coordinate in the scene image is determined by the following equations (3) to (9): [x2,y2,z2]=K -1 ·[u′,v′,1](3) θ=arctan(r / f) (6) i d =θ(1+d1θ 2 +d2θ 4 +d3θ 6 +d4θ 8 (7) x1=(θ d / r)a,y1=(θ d / r)b(8) [u″,v″,1]=K·[x1,y1,1](9) Among them, K corresponds to the camera matrix in the direction; u ′ and v′ are pixel coordinates in the dedistorted image; x2, y2 and z2 are unnormalized camera coordinates; a and b are normalized camera coordinates; r is the normalized incident radius; θ is the incident angle after dedistortion processing; f is the focal length of the camera used to capture the scene image, which defaults to 1; θ d is the incident angle without dedistortion processing, d1, d2, d3 and d4 are distortion parameters respectively; x1 and y1 are the normalized camera coordinates corresponding to the scene image; u″ and v″ are the pixel coordinates of the scene image.

6. The method according to claim 1, characterized in that The texture data includes y channel data, u channel data and v channel data corresponding to the pixel points in the scene image; and converting the texture data corresponding to the scene image in the video memory into target color data includes: The y channel data, u channel data and v channel data corresponding to the pixel point are converted into the r channel data, g channel data and b channel data corresponding to the pixel point by using the following equations (10) to (13): y_t=m1*(y-–m2)(10) r=y_t+m3*v(11) g=y_t-m4*u-m5*v(12) b=y_t+m6*u(13) Among them, y, u and v respectively represent the y channel data, u channel data and v channel data corresponding to the pixel point, and y_t is an intermediate parameter in the conversion process; r, g and b respectively represent the r channel data, g channel data and b channel data corresponding to the pixel point; m1, m2, m3, m4, m5 and m6 are all transformation parameters used when converting y channel data, u channel data and v channel data into r channel data, g channel data and b channel data.

7. The method according to any one of claims 1, 3 to 6, characterized in that: The step of determining, for each of the scene images, a correspondence between a pixel position in a target overhead view image to be generated and each pixel position in the scene image, and converting texture data corresponding to the scene image in the video memory into target color data to obtain a sub-overhead view image corresponding to the scene image, comprises: Determining an initial sub-top view image corresponding to the scene image based on a correspondence between a pixel position in the target top view image and each pixel position in the scene image, and the target color data; The invalid portion of the initial sub-top-view image is removed by using the image mask corresponding to the scene image to obtain the sub-top-view image corresponding to the scene image.

8. The method according to claim 1, characterized in that: The step of combining the sub-top-view images corresponding to the respective scene images through the shader program to obtain the target top-view image comprises: The texture data of the sub-top-view images corresponding to each of the scene images are added together through the shader program to obtain the target top-view image.

9. The method according to any one of claims 1 to 6 or claim 8, characterized in that: The method further comprises: Draw reversing guide lines and virtual radar walls using the target drawing tool; The reversing auxiliary line and the virtual radar wall are covered in the target top view image.

10. An image processing device, characterized in that: The device comprises: An image acquisition module, used to acquire scene images captured by multiple cameras; the multiple cameras are respectively deployed at different positions of the target vehicle body, and are used to capture scene images at different positions around the target vehicle; A texture conversion module, used for converting each of the scene images into corresponding texture data, and saving the texture data into a video memory; A top-view conversion module is used to determine, for each of the scene images, a correspondence between a pixel position in a target top-view image to be generated and each pixel position in the scene image through a shader program running on a graphics processor, and to convert texture data corresponding to the scene image in the video memory into target color data to obtain a sub-top-view image corresponding to the scene image; The bird's-eye view generation module is used to combine the sub-bird's-eye view images corresponding to each of the scene images through the shader program to obtain the target bird's-eye view image.

Images