Image processing method and device

Abstract

The invention discloses an image processing method and an image processing device, which are suitable for converting an image acquired by a vehicle-mounted camera into a target image. The image processing method comprises the following steps: determining a grid division mode and a corresponding weight distribution mode according to an installation posture and an acquisition view angle of a vehicle-mounted camera; dividing the image by using a grid division mode so as to divide the image into a ground grid and a preset number of ground grids, wherein the preset number is related to the grid division mode; and distributing a weight corresponding to each grid by using a weight distribution mode, and determining a target feature parameter of the target image in combination with the feature parameter corresponding to each pixel in the grid. The weight of each window is determined by the importance of the window in a final application scene, and the problem of imbalance existing in image display of a conversion view angle and control (such as automatic exposure and automatic white balance) depending on regional statistics in a traditional statistical window division mode is solved.

Description

Technical Field

[0002] The present invention relates to the field of autonomous driving, and in particular to a method and device for processing images captured by a vehicle-mounted camera. Background Art

[0004] In recent years, autonomous driving technology has developed rapidly, and visual perception technology has played a vital role in intelligent driving, such as automatic parking, surround stitching, and planning control. Depending on the needs of the application scenario, the vehicle will be equipped with different numbers of cameras at specific installation locations.

[0005] For example, the on-board panoramic reversing image system requires that at least one camera be installed in the front, rear, left and right directions of the vehicle. These cameras are used to collect images around the vehicle, and when displayed, the images around the vehicle are displayed from a bird's-eye view, that is, a bird's-eye view image. Obviously, these on-board cameras do not directly shoot the top-down image, but need to convert the front image, rear image, left image and rear side image shot by multiple cameras into a bird's-eye view image and stitch them to generate a panoramic image. It can be imagined that when the actual image shot by the on-board camera is converted to other perspectives for display, the area of ​​interest in the image will be very different due to the different perspectives. For example, the near area that accounts for a large proportion in the front view actually accounts for a small proportion in the bird's-eye view, and the distant road area that accounts for a small proportion in the front view actually accounts for a large proportion in the bird's-eye view. Therefore, when the perspective is converted to display, if the statistical window (rectangle or a combination of multiple rectangles) is set based on the original shooting perspective, it will often cause the proportion of the bird's-eye view to be abnormal. At the same time, when the lighting environment is complex, such as when there are car lights nearby and street lights in the distance, if conventional methods are used for automatic exposure statistics or white balance statistics, the color and exposure in the bird's-eye view will be more inclined to the nearby area that accounts for a larger proportion in the front view, while ignoring the distant area that accounts for a large proportion in the bird's-eye view.

[0006] Even in image display that does not require a change in viewing angle, such as a single-lens reversing image display, if there is a multi-light source environment such as backlighting or night vision, the on-board camera needs to be able to flexibly adjust the white balance settings to obtain accurate color and exposure. These situations are also closely related to the way the statistical window is divided.

[0007] Traditional statistics windows such as Figure 1 or Figure 2 As shown, Figure 1 The uniformly planned rectangular statistical window division method is shown. Figure 2 The rectangular statistical window division method of different resolution plans is shown, where Figure 2 The window division method shown can also be Figure 1Obviously, as mentioned above, the traditional statistical window division method is not conducive to the image display of the conversion perspective, nor is it conducive to automatic exposure statistics or white balance statistics.

[0008] In order to solve the above problems, the present invention proposes an image processing method and a device thereof. Summary of the invention

[0010] The present invention proposes an image processing method and a device thereof, which can set the division method and weight distribution method of statistical windows accordingly based on the application scenario of the vehicle-mounted camera, so that the weight of each window is determined by its importance in the final application scenario.

[0011] According to one aspect of the present invention, the present invention provides an image processing method, which is suitable for converting an image captured by a vehicle-mounted camera into a target image. According to one aspect, the image processing method includes: determining a grid division method and a corresponding weight distribution method according to the installation posture and acquisition angle of the vehicle-mounted camera; dividing the image by the grid division method to divide the image into a ground grid and a preset number of ground grids, the preset number being related to the grid division method; and using a weight distribution method to allocate a weight corresponding to each grid and determine the target feature parameters of the grid presented in the target image in combination with the feature parameters of each grid, and the target feature parameters of all grids constitute the target feature parameters of the target image.

[0012] According to one aspect, the grid division method includes M first dividing lines L extending along the first direction and arranged in sequence along the second direction. 1i and N-1 second dividing lines L extending in a direction intersecting with the first direction 2j , N-1 second dividing lines L 2j The images are not intersected and extend in different directions, and are arranged in sequence along a first direction, wherein i∈[1,M], j∈[1,N-1], M≥2, N≥2, the first direction and the second direction are perpendicular, and the image is divided into a ground grid and a preset number of ground grids by using a grid division method, including: using a first dividing line L 11 Divide the image into a ground grid and a first ground grid; and use a first segmentation line L 1k And N-1 second dividing lines L 2j The first ground grid is divided into M*N ground grids, where k∈[2,M] and M*N is a preset number.

[0013] According to one aspect, the weight allocation method is a manual allocation method, and the weight value of each ground grid is a positive number. Among the M*N ground grids, the weights of the M ground grids arranged in sequence along the second direction in any column decrease successively, and the weights of the N ground grids arranged in sequence along the first direction in any row decrease successively from the middle grid to the grids on both sides, and the weight of the ground grid is the smallest.

[0014] According to one aspect, an extending direction of at least one second segmentation line is parallel to an extending direction of a lane line in the image.

[0015] According to one aspect, before determining the grid division method and its corresponding weight distribution method according to the installation posture and acquisition angle of the vehicle-mounted camera, it also includes: preprocessing the image to obtain the area of ​​interest of the image, the preprocessing at least includes removing invalid areas in the image; and dividing the image using the grid division method to divide the image into a ground grid and a preset number of ground grids, including: taking the area of ​​interest as the division object of the grid division method, dividing the area of ​​interest using the grid division method to divide the area of ​​interest into a ground grid and a preset number of ground grids.

[0016] According to one aspect, the invalid area is characterized by an upper boundary, a lower boundary, a left boundary and a right boundary, and removing the invalid boundaries in the image includes: obtaining the size, upper boundary, lower boundary, left boundary and right boundary of the image based on the installation posture of the vehicle-mounted camera; and removing the upper boundary, lower boundary, left boundary and right boundary area from the image to form an area of ​​interest.

[0017] According to one aspect, the target image is a bird's-eye view image, and the target feature parameter is a certain statistic of the image content, such as brightness statistics and color statistics. Before determining the grid division method and its corresponding weight distribution method according to the installation posture and acquisition angle of the vehicle-mounted camera, it also includes: obtaining the internal parameters corresponding to the image and the corresponding external parameters that can be mapped to the bird's-eye view; using the grid division method corresponding to the installation posture and acquisition angle of the vehicle-mounted camera to divide the image sample to obtain the preset number of ground grids; for each ground grid, determining the mapping relationship between the image coordinate system of the image and the target coordinate system of the target image based on the internal and external parameters; using the mapping relationship to calculate the mapping grid of the ground grid in the bird's-eye view; and using the relative relationship between the area of ​​the mapping grid and the ground grid to determine the weight of the ground grid, the weights of all ground grids and the weight of the ground grid constitute the weight distribution method.

[0018] According to one aspect, using a mapping relationship to calculate a mapping grid of a ground grid in a bird's-eye view image includes: dividing the ground grid into a plurality of quadrilateral pixel areas; and for each pixel area in the ground grid, using a mapping relationship between an image coordinate system and a target coordinate system, calculating the coordinates of four vertices of the pixel area at four mapping points in the target coordinate system, wherein the quadrilateral surrounded by the four mapping points constitutes a mapping area corresponding to the pixel area; using a relative relationship between the areas of the mapping grid and the ground grid to determine the weight of the ground grid includes: for each pixel area, calculating the area of ​​the pixel area and the area of ​​its corresponding mapping area; using the ratio of the area of ​​the mapping area corresponding to each pixel area to the area of ​​the pixel area as the weight of the pixel area; and using the average value of the weights of all pixel areas of the ground grid as the weight of the ground grid.

[0019] According to one aspect, the quadrilateral pixel area is a unit square area centered at any pixel point.

[0020] According to one aspect, the image processing method further includes: manually setting a weight of a ground grid, the weight of the ground grid being smaller than a weight of any ground grid.

[0021] According to one aspect, the characteristic parameter is a white balance statistic, and a weight corresponding to each grid is assigned using a weight assignment method and a target characteristic parameter presented by the grid in the target image is determined in combination with the characteristic parameters of each grid, including: calculating a target baseline white balance statistic of the target image based on the baseline white balance statistics of all grids and their corresponding weights; and adjusting the white balance statistics of all grids based on the target baseline white balance statistics to calculate a white balance gain to generate a target image.

[0022] According to one aspect, the characteristic parameter is a brightness statistic, and a weight corresponding to each grid is assigned using a weight distribution method and a target characteristic parameter presented by the grid in the target image is determined in combination with the characteristic parameters of each grid, including: calculating a target baseline brightness statistic of the target image based on the baseline brightness statistics of all grids and their corresponding weights; and adjusting the exposure statistics of all grids based on the target baseline brightness statistics to calculate an automatic exposure control amount and an automatic gain control amount to generate a target image.

[0023] According to another aspect of the present invention, an image processing device is also disclosed, including: at least one memory for storing a computer program; and at least one processor connected to the at least one memory, the at least one processor being used to execute the computer program, and when the computer program is executed, an image processing method as described in any of the foregoing is implemented.

[0024] According to another aspect of the present invention, a computer storage medium is also disclosed for storing a computer program, and when the computer program is executed, any of the aforementioned image processing methods is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a uniform rectangular grid division method according to the prior art;

[0028] Figure 2 A schematic diagram of a non-uniform rectangular grid division method according to the prior art;

[0029] Figure 3 A schematic diagram of a flow chart of an image processing method in an embodiment according to one aspect of the present invention;

[0030] Figure 4 A schematic diagram of a collection viewing angle of a rear view camera in an embodiment according to one aspect of the present invention;

[0031] Figure 5 A schematic diagram of a grid division method in an embodiment according to one aspect of the present invention; Figure 6 A schematic diagram of a grid division method in an embodiment according to one aspect of the present invention;

[0032] Figure 7 A schematic diagram of a grid division method in an embodiment according to one aspect of the present invention;

[0033] Figure 8 A schematic diagram of a flow chart of a mesh division step in an embodiment according to one aspect of the present invention;

[0034] Fig. 9 A schematic flow chart of a pre-processing step in an embodiment according to one aspect of the present invention;

[0035] Fig.10 A schematic diagram of a mapping relationship in an embodiment according to one aspect of the present invention;

[0036] Fig.11 It is a flow chart of a method for determining a weight allocation method in an embodiment according to one aspect of the present invention;

[0037] Fig.12 FIG. 4 is a schematic block diagram of an image processing device in an embodiment according to another aspect of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] According to one aspect of the present invention, there is provided an image processing method suitable for converting an image captured by a vehicle-mounted camera into a target image.

[0041] Among them, the vehicle-mounted camera refers to a manufacturer-configured or post-installed photo-taking device or video-taking device installed on the vehicle for capturing images outside the vehicle.

[0042] The target image refers to the application scenario of the vehicle camera, specifically the final presentation image of the image collected by the vehicle camera. For example, the bird's-eye view image presented in the panoramic reversing image, the image directly displayed in the reversing image of a single camera, can be concretely understood as the picture or video that is finally presented to the occupants of the vehicle through the vehicle display device.

[0043] Different application scenarios of vehicle cameras will cause the target image to have different key image parameters. For example, in a bird's-eye view image, the size or proportion of the lanes in the image and the people or objects on the lanes that affect the vehicle's driving, the color and exposure in the rear view image, etc. Therefore, vehicle cameras in different application scenarios may have different target images, and the key image parameters of the target images may also be different, such as the exposure statistics in automatic exposure control and the white balance statistics in automatic white balance control.

[0044] Specifically, Figure 3 FIG. 2 shows a schematic diagram of steps of an image processing method in an embodiment. Figure 3 As shown, the image processing method may include steps S310 to S330.

[0045] Among them, step S310 is: determining the grid division method and its corresponding weight distribution method according to the installation posture and acquisition viewing angle of the vehicle-mounted camera.

[0046] The installation posture of the vehicle camera refers to the installation position of the vehicle camera on the vehicle, the roll angle, pitch angle and heading angle of the vehicle camera and other parameters that affect the shooting area and shooting angle of the vehicle camera.

[0047] The acquisition angle refers to the field of view of the vehicle camera, which is usually a spatial angle emitted from the installation position of the vehicle camera as the origin, such as Figure 4 The acquisition angle of view Q1 of the rear-view vehicle camera in a specific embodiment shown. The acquisition angle of view is related to the focal length of the vehicle camera, and different acquisition angles of view may affect the range of the effective area in the image captured by the vehicle camera. Taking the side-view camera as an example, part of the vehicle body often appears in the images captured by some side-view cameras, while the driver only needs to know the boundary of the vehicle and does not need to see the vehicle body. Therefore, the part of the vehicle body captured by the camera is usually as small as possible but cannot be absent, and the acquisition angle of view and the installation posture jointly determine the area size of the vehicle body that appears in the image captured by the vehicle camera. In the preferred grid division method, the vehicle body part is usually regarded as an invalid area. This is because in some application scenarios, such as the commonly used bird's-eye view, the vehicle body usually appears as a fixed virtual image instead of displaying the actual captured image.

[0048] The grid division method refers to the division of the image into statistical windows. The weight allocation method refers to assigning a weight to each statistical window.

[0049] It can be understood that the installation posture and acquisition angle of the vehicle-mounted camera determine the effective range of the image captured by the vehicle-mounted camera and the image presentation within the range. Therefore, regardless of whether the vehicle-mounted camera is correctly installed or whether there are changes during actual operation, the effective range of the currently captured image and the image presentation within the range can be determined based on the current installation posture and acquisition angle of the vehicle-mounted camera, and then the grid division method can be determined in a preset manner.

[0050] Optionally, in other embodiments, a preprocessing step may be included before step S310, in which images captured by the vehicle-mounted camera at different postures or viewing angles are processed into preprocessed images that meet the implementation requirements of the grid division method.

[0051] Step S320 is: dividing the image by using a grid division method to divide the image into a ground grid and a preset number of ground grids.

[0052] In the process of implementing the present invention, the present application found that one of the problems with the traditional window statistics method is that all areas in the image are evenly divided and / or all windows are given the same weight. The present application at least distinguishes the grid division method and weight value of the above-ground part and the ground part in the image.

[0053] The ground grid refers to the sky area in the image or the area above the ground and above the horizon area in the imaging. It can be understood that the horizon refers to the actual operating ground of the vehicle. Depending on the terrain, the horizon in the image will be presented as a relatively flat line, or it may be presented in the form of irregular lines. A person skilled in the art can set a straight line close to the ground plane as the dividing line between the ground grid and the ground grid based on the acquisition angle of the vehicle-mounted camera, which is called the ground plane dividing line. Generally, the ground plane dividing line is parallel to the lower edge line of the image. Preferably, the ground plane dividing line can be set to any point in the interval from the highest point of the horizon to the lowest point of the horizon and a straight line parallel to the lower edge line, and can even be set to the interval from slightly higher than the highest point of the horizon to slightly lower than the lowest point of the horizon. It can be understood that most of the sky is above the ground plane, and the area closer to the ground plane is generally far away from the vehicle, so the ground plane dividing line does not need to be very accurate.

[0054] Then, the area above the ground plane dividing line constitutes a ground grid, and further dividing lines can be set below the ground plane dividing line to divide more grids to form multiple ground grids. The number of ground grids is directly related to the grid division method, so the preset number corresponds to the grid division method one by one.

[0055] The multiple segmentation lines included in the grid division method can be stored in the form of coordinates. Taking the image coordinate system where the image is located as the coordinate system of the grid division method as an example, the multiple segmentation lines that divide the grid are stored in the form of coordinates of each pixel point on it or the coordinates of the pixel points at both ends of the segmentation line. When segmenting, the grid division can be completed on the image by connecting the pixel points corresponding to the stored coordinates. The focus of this application is on the division shape of the grid, and it does not limit the implementation method of the division action. Those skilled in the art can understand that the methods that can achieve the grid division results described in this application should be covered within the scope of protection of this application.

[0056] Figure 5 FIG. 2 shows a schematic diagram of a mesh division shape in an embodiment. Figure 5 As shown, y0-y3 is the ground plane segmentation line that divides the image into the ground grid Win0 and the ground part, and the ground part is further divided into 3*3 grids Win1, Win2, Win3, Win4, Win5, Win6, Win7, Win8 and Win9 by y1-y4, y2-y5, x0-x2 and x1-x3, wherein the segmentation lines y1-y4 and y2-y5 are parallel to the lower edge of the image, and the segmentation lines x0-x2 and x1-x3 have an angle with the extension direction of the lower edge and are not equal to 90°. Preferably, the segmentation lines x0-x2 and x1-x3 extend along the lane line.

[0057] Understandably, Figure 5The rear view angle is divided into left, middle and right horizontally, and into near, middle distance and far distance vertically. The grids Win1, Win2, Win3, Win4, Win5, Win6, Win7, Win8 and Win9 represent far left, far middle, far right, middle left, middle middle, middle right, near left, near middle and near right respectively.

[0058] Combination Figure 5 The embodiment shown, Figure 6 Figure 2 shows a schematic diagram of grid division. Figure 6 As shown, the grid division method includes M first dividing lines L 1i and N-1 second dividing lines L 2j , i∈[1,M], j∈[1,N-1], M≥2, N≥2. Among them, there are M first cutting lines L 1i Extending along the first direction x and arranged in sequence along the second direction y; N-1 second cutting lines L 2j do not cross each other and extend in different directions. At the same time, any second dividing line L 2j The extending direction intersects with the first direction x, and the first direction x is perpendicular to the second direction y.

[0059] It should be noted that "non-intersecting" means: there is no intersection between multiple line segments; "different extension directions" means: with any direction as the reference axis, the rotation angles of the extension lines of multiple line segments relative to the reference axis are not equal; "arranged in sequence" means: with a straight line passing through all the segmentation lines as the reference axis, the intersections of all the segmentation lines with the reference axis are arranged in sequence according to the extension direction of the segmentation lines; for example, "N-1 second segmentation lines are arranged in sequence along the first direction x" means: with reference to Figure 6 , taking a straight line Lx extending in a direction parallel to the first direction x and passing through all the second cutting lines as the reference axis, then the second cutting line L 21 ~L 2N-1 The intersection points with the reference axis Lx are arranged in sequence.

[0060] Preferably, at least one second cutting line L 2j The extension direction is parallel to the extension direction of any lane line in the image; when the lane lines in the image are symmetrical, N-1 second dividing lines are symmetrically distributed, and if N-1 is an odd number, the N / 2 second dividing line can extend along the symmetry axis. Based on the characteristics of the second dividing line, in some embodiments, the second dividing line can also be drawn by image recognition.

[0061] It can be understood that when the second dividing line extends along the lane line, the grid division method is closest to driving needs. In this application, "the extension direction is parallel to the extension direction of the lane line" means that the angle of the extension direction relative to the extension direction of the lane line is within an acceptable error range. That is, when dividing the grid, the second dividing line can be roughly parallel to the lane line.

[0062] Preferably, in the grid division method of the front view or rear view camera, M can be set to 3 and N can be set to 3.

[0063] Preferably, the basic settings of the grid division methods of the multiple vehicle-mounted cameras can be the same, such as 3*3. However, in the grid division method of the side view camera, the 3*3 grid setting is equivalent to 2*2 through special parameter configuration.

[0064] Figure 7 A schematic diagram showing the mesh division shape of a side-view camera in one embodiment is shown, Figure 7 The logo and Figure 5 The corresponding symbols in represent the same meaning and can be understood by comparison. Figure 7 As shown, a schematic diagram of a side front view is given here, where x1 and x2 are set outside the image range, and some grids such as win3 almost do not exist. It can be understood that in other embodiments, y2-y5 and x0-x2 can also be set outside the image range, and the ground part will be divided into 2*2 ground grids by y1-y4 and x1-x3.

[0065] Figure 8 Shown based on Figure 6 The schematic diagram of the meshing process of the meshing method shown in FIG. Figure 8 As shown, step S320 may include steps S321~S322.

[0066] Among them, step S321 is: using the first cutting line L 11 Divide the image into a ground grid and a first ground grid.

[0067] The ground grid refers to the area above the horizon dividing line. The first ground grid refers to another part of the grid divided by the horizon dividing line. It should be noted that in the present invention, the first dividing line L 11 Refers to the dividing line closest to the sky part of the image, that is, the horizon dividing line.

[0068] Step S322 is: using a plurality of first cutting lines L 1k And N-1 second dividing lines L 2j The first ground grid is divided into M*N ground grids, where k∈[2,M].

[0069] The aforementioned preset number is M*N ground grids.

[0070] Further, Figure 5 The embodiment shown clearly sets a region of interest (ROI). Figure 5 As shown, in some embodiments, a preprocessing step may be set before step S310: preprocessing the image captured by the vehicle-mounted camera to obtain a region of interest ROI of the image. The preprocessing at least includes removing invalid boundaries in the image. These invalid boundaries are usually areas where the vehicle body or the lens frame blocks the viewing angle.

[0071] Fig. 9 FIG. 2 shows a schematic diagram of a sub-process of a pre-processing step in an embodiment. Fig. 9 As shown, the preprocessing step includes steps S910 to S920.

[0072] Wherein, step S910 is: based on the installation posture of the vehicle-mounted camera, the size and upper boundary T of the image are obtained. U , lower boundary T B , left boundary T Z and right boundary T Y .

[0073] Step S920 is: remove the upper boundary, the lower boundary, the left boundary and the right boundary from the image to form a region of interest ROI.

[0074] It should be noted that the sidelines in this application refer to the four sides of a rectangular image, and the border refers to a long strip area extending from one side of the image to the opposite side and having a certain width.

[0075] Furthermore, the preprocessing step may also include other conventional preprocessing steps in the art, such as viewing angle correction of a vehicle-mounted camera, etc.

[0076] It can be understood that in the embodiment covering the pre-processing step, the objects of the image processing methods S310 to S330 should all be replaced by the region of interest ROI.

[0077] Furthermore, if Figure 3 As shown, after the grid division is completed, step S330 is: using the weight distribution method to allocate the weight corresponding to each grid and combining the feature parameters of each grid to determine the target feature parameters presented by the grid in the target image, and the target feature parameters of all grids constitute the target feature parameters of the target image.

[0078] The weight distribution method can be manually distributed by technicians in this field based on experience; or it can be reversely assigned to the grid weights in the image corresponding to the target image according to the characteristics of the target image; or it can use known corresponding images and target images as samples, and use mathematical calculations or machine learning methods to determine the weight distribution method of samples with known grid division methods.

[0079] The following combination Figure 6 The meshing method shown in Figure 2 illustrates the manual assignment method. Figure 6 As shown, M*N ground grids are represented in the form of a matrix, and the j-th column grid arranged along the y direction is Q 1j ~Q Mj , the i-th row of grids arranged along the x direction is Q i1 ~Q iN , where i∈[1,M], j∈[1,N-1], Q 1j is the grid closest to the horizon dividing line. Then, in the jth column grid Q 1j ~Q Mj In the grid, usually Q 1j ~Q Mj The weights of the M ground grids arranged along the second direction decrease in sequence; in the i-th row of grids Q i1 ~Q iN In the grid, usually Q i1 ~Q iN The weights decrease from the middle to the sides, that is, the weights of the N ground grids arranged along the first direction decrease from the middle grid to the grids on both sides; the weight of the ground grid Q0 is the smallest.

[0080] by Figure 5 Taking the grid division method shown as an example, the weight of window Win0 can be set to 1, the weight of window Win1 can be set to 20, the weight of window Win2 can be set to 16, the weight of window Win3 can be set to 20, the weight of window Win4 can be set to 12, the weight of window Win5 can be set to 8, the weight of window Win6 can be set to 12, the weight of window Win7 can be set to 9, the weight of window Win8 can be set to 6, and the weight of window Win9 can be set to 9.

[0081] It is understandable that based on the application scenarios of vehicle-mounted cameras, the cost increase brought by high computing power, and the visual requirements of the naked eye, reasonable M and N values ​​can be set and should not be infinitely expanded. For vehicle-mounted front-view, rear-view, and side-view cameras, the typical configuration is M=N=3, usually M≤5, N≤5.

[0082] The following is an example of how to determine the weight distribution method through calculation in combination with a specific application scenario.

[0083] Taking the bird's-eye view image as the target image as an example, the conversion relationship between the image coordinate system, the vehicle coordinate system and the world coordinate system is common knowledge in the field and will not be described in detail below. Technical personnel in the field can obtain theoretical and detailed descriptions by searching the Internet or consulting textbooks.

[0084] Those skilled in the art will understand that after the vehicle-mounted camera is installed on the vehicle, its position and posture relative to the vehicle body coordinate system (or world coordinate system) are determined. In fact, for cameras installed in advance by car manufacturers, after the car model is designed, the position and posture of the camera relative to the vehicle body or world coordinate system are relatively determined, and its design parameters basically meet the use of the present invention.

[0085] Assuming camera intrinsics , assuming that the camera distortion has been removed. Points on the road Assume that the rotation matrix and translation vector of the camera extrinsic parameters are and The pixel position of the point on the road surface imaged by the camera is , in .in, and are the horizontal and vertical focal lengths of the camera in pixels, respectively. and are the principal point coordinates of the camera, Points The coordinates in the world coordinate system, r 11 、r 12 、r 13 、r 21 、r 22 、r 23 、r 31 、r 32 and r 33 They are the nine elements of the rotation matrix (3x3) of the camera extrinsic parameters, They are the three components of the camera’s extrinsic translation vector, p u Points on the road The pixels that form an image in the camera. is the coordinate of the pixel in the image coordinate system (1 means that the image coordinate system is a two-dimensional coordinate system).

[0086] when When (considering only points on the ground (world coordinate system z=0)), it is simplified to:

[0087] (1)

[0088] If we consider the distortion effect of the lens, the actual ground point The coordinates projected to the pixel in the camera are:

[0089] (2)

[0090] It is the mapping relationship of lens distortion.

[0091] From the above geometric relationship, we can get the actual mapping from ground points to image pixel coordinates.

[0092] In the embodiment using the bird's-eye view image as the target image, according to the aforementioned geometric relationship, a plane to be finally used is selected to be defined as the world coordinate system. flat.

[0093] , .

[0094] Considering that this mapping is plane-to-plane, it can also be expressed in reverse:

[0095] (3)

[0096]

[0097] For a small face element du dv in an image, its corresponding area on the ground plane is dx dy, and the area magnification of this face element in the process of projecting from the image to the ground plane corresponds to its importance.

[0098] (4)

[0099] However, there may not be an analytical expression for the description of lens distortion, so it may be difficult to directly use function differentiation to solve r(p). In practice, it can be solved by numerical methods. Fig.10 As shown, for the pixel point in the image pixel plane ( The unit square centered , its four vertices can be projected to their corresponding points in the target plane (ground plane) after the above transformation , , and The ratio of the area of ​​the quadrilateral formed by these four points to the area of ​​the unit square in the pixel plane can be regarded as the center point of the unit square in the pixel plane ( It should be pointed out that this ratio is ultimately reflected in the weight of the statistical pixel points, and the weight itself is also relative, so the area unit can be ignored for the area within the target plane. There are many ways to calculate the area of ​​a quadrilateral, given that the coordinates of the four vertices are known, including but not limited to using the coordinates of the four points to calculate the length of each side and one of the diagonals, and then using Heron's formula to calculate the area of ​​the two triangles divided by the diagonal to synthesize the area of ​​the quadrilateral.

[0100] Combined with the above calculation process, Fig.11 FIG. 1 is a flow chart showing a method for determining a weight distribution method for a grid division method in an embodiment. Fig.11 As shown, the step of determining a weight distribution method of a grid division method may include steps S101 to S105.

[0101] Wherein, step S101 is: obtaining the internal parameters of the image captured by the vehicle-mounted camera and the corresponding external parameters that can be mapped to the bird's-eye view.

[0102] Step S102 is: determining a mapping relationship between an image coordinate system of an image captured by the vehicle-mounted camera and a target coordinate system of a target image based on an internal parameter and an external parameter.

[0103] Step S103 is: dividing the image samples using the grid division method corresponding to the installation posture and acquisition viewing angle of the vehicle-mounted camera to obtain a preset number of ground grids.

[0104] Different installation postures and acquisition viewing angles may correspond to different grid division methods. Therefore, it is necessary to determine the weight distribution method for the image samples of each grid division method.

[0105] Step S104 is: for each ground grid, using the mapping relationship between the image coordinate system and the target coordinate system of the target image, a mapping grid of the ground grid in the bird's-eye view image is calculated.

[0106] In the bird's-eye view image, the target coordinate system is the earth's coordinate system.

[0107] Specifically, step S104 can be refined as follows: dividing the ground grid into a plurality of quadrilateral pixel regions, and for each pixel region in the ground grid, using the mapping relationship between the image coordinate system and the target coordinate system, calculating the coordinates of the four vertices of each pixel region in the target coordinate system. The quadrilateral surrounded by the four mapping points constitutes the mapping region corresponding to the pixel region.

[0108] It can be understood that in order to save computing power, each grid can be divided into a certain number of pixel regions. When the computing power is sufficient, each pixel point can be set as a pixel region.

[0109] Step S105 is: for each ground grid, the weight of the ground grid is determined by using the relative relationship between the areas of the mapping grid and the ground grid. The weights of all ground grids and the weight of the ground grid constitute a weight distribution method corresponding to the grid division method.

[0110] Corresponding to the refinement step of step 104, step S105 can be refined as follows: for each pixel area, calculate the area of ​​the pixel area and the area of ​​its corresponding mapping area; take the ratio of the area of ​​the mapping area corresponding to each pixel area to the area of ​​the pixel area as the weight of the pixel area; and take the average value of the weights of all pixel areas in the ground grid as the weight of the ground grid.

[0111] It can be understood that the step of determining a weight distribution method for a grid division method also includes setting the weight of the ground grid. The simplest method is manual setting, and the weight of the ground grid is usually set to the minimum value, that is, less than the weight of any ground grid.

[0112] The above description uses the bird's-eye view image as an example to explain in detail the method for determining the weight allocation method in this application scenario. In specific application scenarios, such as automatic exposure control (AEC) and automatic white balance (AWB), the weight allocation method and grid division method may be different, but they can basically be obtained based on the comparison between the parameter characteristics of the target image in the sample and the corresponding parameter characteristics in the acquired image.

[0113] Taking the white balance application scenario as an example, the corresponding characteristic parameter is the white balance statistic, and the aforementioned step S130 can be correspondingly refined as follows: calculating the target baseline white balance statistic of the target image based on the baseline white balance statistics of all grids and their corresponding weights; and adjusting the white balance gains of all grids based on the target baseline white balance statistics to generate the target image.

[0114] Taking the exposure control application scenario as an example, the corresponding characteristic parameter is the exposure statistic, and the aforementioned step S130 can be correspondingly refined as follows: calculating the target baseline exposure statistic of the target image based on the baseline exposure statistics of all grids and their corresponding weights; and adjusting the exposure statistics of all grids based on the target baseline exposure statistics to calculate the automatic exposure and automatic gain control amounts to generate the target image.

[0115] It is understandable that in a complex light source environment, such as a parking lot environment, the rearview camera often has a reddish color in the near distance (caused by the brake lights) and a bluish-green color in the far distance (caused by the lighting). If conventional statistical methods are used, the near distance occupies a large area, and the red brake lights in the near distance are mainly used for statistics. Obviously, using the near ground as the white balance reference will cause the display screen to be weird. However, after being projected to the ground perspective, the residual green light in the distance will occupy a large area, and the color in the bird's-eye view is very abnormal. After adopting the grid division method of the present invention, the weight of the far distance is increased, and the AWB result will tend to use the far ground as the white balance reference, and it will be relatively normal after being projected to the bird's-eye view.

[0116] The descriptions of the processes corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0117] According to another aspect of the present invention, an image processing device is provided.

[0118] Fig.12 FIG. 1 is a schematic block diagram of an image processing device 120 in an embodiment. Fig.12 As shown, the image processing device 120 includes at least one memory 121 ( Fig.12 For example, one) and at least one processor 122 ( Fig.12 Taking one as an example), at least one memory 121 is used to store computer programs, and at least one processor 122 is used to run the programs stored in the at least one memory 121 to implement the image processing method described above.

[0119] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0120] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0121] According to yet another aspect of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium is used to store a computer program. When the computer program is directly or indirectly executed, the image processing method described above can be implemented.

[0122] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0123] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0124] In this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0125] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only exemplary and can be divided into different functional units from a functional perspective.

[0128] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0129] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, suitable for converting an image captured by a vehicle-mounted camera into a target image, characterized in that: include: Determine a grid division method and a corresponding weight distribution method according to the installation posture and acquisition viewing angle of the vehicle-mounted camera; Dividing the image by using the grid division method to divide the image into a ground grid and a preset number of ground grids, wherein the preset number is related to the grid division method; as well as The weight corresponding to each grid is allocated by the weight allocation method and the target feature parameters of each grid are determined in combination with the feature parameters of each grid. The target feature parameters of all grids constitute the target feature parameters of the target image.

2. The image processing method according to claim 1, characterized in that: The grid division method includes M first dividing lines L extending along the first direction and arranged in sequence along the second direction 1i and N-1 second dividing lines L extending in a direction intersecting with the first direction 2j , the N-1 second dividing lines L 2j They do not cross each other and extend in different directions, and are arranged in sequence along the first direction, wherein: i ∈[1,M], j∈[1,N-1], M≥2, N≥2, the first direction is perpendicular to the second direction, and the step of dividing the image into a ground grid and a preset number of ground grids by using a grid division method comprises: Using the first dividing line L 11 dividing the image into an above-ground grid and a first ground grid; and Using the first dividing line L 1k And N-1 second dividing lines L 2j The first ground grid is divided into M*N ground grids, where: k ∈[2,M], M*N is the preset number.

3. The image processing method according to claim 2, characterized in that: The weight allocation method is a manual allocation method, and the weight value of each ground grid is a positive number. Among the M*N ground grids, the weights of the M ground grids arranged in sequence along the second direction in any column decrease successively, and the weights of the N ground grids arranged in sequence along the first direction in any row decrease successively from the middle grid to the grids on both sides, and the weight of the ground grid is the smallest.

4. The image processing method according to claim 2, wherein: An extension direction of at least one of the second dividing lines is parallel to an extension direction of a lane line in the image.

5. The image processing method according to claim 1 or 2, characterized in that: Before determining the grid division method and the corresponding weight distribution method according to the installation posture and the acquisition angle of the vehicle-mounted camera, the method further includes: Preprocessing the image to obtain a region of interest in the image, wherein the preprocessing at least includes removing an invalid region in the image; and The method of dividing the image into a ground grid and a preset number of ground grids by using a grid division method comprises: The region of interest is used as a division object of the grid division method, and the region of interest is divided by the grid division method to divide the region of interest into a ground grid and a preset number of ground grids.

6. The image processing method according to claim 5, characterized in that: The invalid area is characterized by an upper boundary, a lower boundary, a left boundary and a right boundary, and removing the invalid boundary in the image includes: Acquire the size of the image, the upper boundary, the lower boundary, the left boundary, and the right boundary based on the installation posture of the vehicle-mounted camera; and The regions of the upper boundary, the lower boundary, the left boundary, and the right boundary are removed from the image to form the region of interest.

7. The image processing method according to claim 1, characterized in that: The target image is a bird's-eye view image, and the target feature parameter is a statistic of the image content. Before determining the grid division method and the corresponding weight distribution method according to the installation posture and the acquisition viewing angle of the vehicle-mounted camera, the method further includes: Obtaining an internal parameter corresponding to the image and a corresponding external parameter that can be mapped to a bird's-eye view; Determine a mapping relationship between an image coordinate system of the image and a target coordinate system of the target image based on the internal and external parameters; Dividing the image sample by using the grid division method corresponding to the installation posture and acquisition angle of view of the vehicle-mounted camera to obtain the preset number of ground grids; For each ground grid, Calculating a mapping grid of the ground grid in the bird's-eye view image using the mapping relationship; and The weight of the ground grid is determined by using the relative relationship between the areas of the mapping grid and the ground grid. The weights of all ground grids and the weights of the ground grids constitute the weight distribution method.

8. The image processing method according to claim 7, characterized in that: The method of calculating the mapping grid of the ground grid in the bird's-eye view image by using the mapping relationship includes: Dividing the ground grid into a plurality of quadrilateral pixel areas; and For each pixel area in the ground grid, using the mapping relationship, the coordinates of four mapping points of the four vertices of the pixel area in the target coordinate system are calculated, and the quadrilateral surrounded by the four mapping points constitutes the mapping area corresponding to the pixel area; Determining the weight of the ground grid by using the relative relationship between the areas of the mapping grid and the ground grid includes: For each pixel region, calculating the area of ​​the pixel region and the area of ​​the corresponding mapping region; taking the ratio of the area of ​​the mapping area corresponding to each pixel area to the area of ​​the pixel area as the weight of the pixel area; and The average value of the weights of all pixel areas of the ground grid is used as the weight of the ground grid.

9. The image processing method according to claim 8, characterized in that: The quadrilateral pixel area is a unit square area centered at any pixel point.

10. The image processing method according to any one of claims 7 to 9, characterized in that: Also includes: The weight of the ground grid is manually set, and the weight of the ground grid is less than the weight of any ground grid.

11. The image processing method according to claim 1, wherein: The characteristic parameter is a white balance statistic, and the weight corresponding to each grid is allocated by a weight allocation method and the target characteristic parameter presented by the grid in the target image is determined by combining the characteristic parameter of each grid, including: Calculating a target reference white balance statistic of the target image based on the reference white balance statistics of all grids and their corresponding weights; and The white balance statistics of all grids are adjusted based on the target reference white balance statistics to calculate the white balance gain of the target image to generate the target image.

12. The image processing method according to claim 1, characterized in that: The characteristic parameter is an exposure statistic, and the weight corresponding to each grid is allocated by a weight allocation method and the target characteristic parameter presented by the grid in the target image is determined by combining the characteristic parameter of each grid, including: Calculating a target baseline exposure statistic of the target image based on the baseline exposure statistics of all grids and their corresponding weights; and The exposure statistics of all grids are adjusted based on the target reference exposure statistics to calculate an automatic exposure control amount and an automatic gain control amount of the target image to generate the target image.

13. An image processing device, characterized in that: include: at least one memory for storing a computer program; as well as At least one processor connected to the at least one memory, the at least one processor is used to execute the computer program, and when the computer program is executed, the image processing method according to any one of claims 1 to 12 is implemented.

14. A computer storage medium for storing a computer program, characterized in that: When the computer program is executed, the image processing method according to any one of claims 1 to 12 is implemented.

Images