Image processing method, device and equipment

Abstract

The invention provides an image processing method, device and equipment, and the method comprises the steps: determining a first proportion value of a first grid based on a target panoramic image, the first proportion value being a proportion value of a shadow region area of a first image region to a total area of the first image region; mapping the plurality of vertexes of the vehicle into a plurality of ground projection points based on the candidate light directions, wherein the plurality of ground projection points form a plurality of shadow surfaces; determining a second proportion value of the first grid based on the plurality of shadow surfaces, the second proportion value being a proportion value of the number of reference sub-grids and the total number of the plurality of sub-grids, the reference sub-grids being sub-grids having an overlapping region with any shadow surface; based on a difference value between the first proportion value and a second proportion value corresponding to each candidate light direction, selecting the candidate light direction with the minimum difference value as a target light direction; and performing brightness compensation on the first image area based on a target brightness compensation value corresponding to the target light direction. According to the technical scheme, the image display effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to an image processing method, device and equipment. Background Art

[0002] With the continuous development of image processing technology, the vehicle panoramic system has also been widely used. The vehicle panoramic system includes multiple cameras installed on the vehicle, such as a camera located in front of the vehicle, a camera located behind the vehicle, a camera located on the left side of the vehicle, a camera located on the right side of the vehicle, etc. During the driving process of the vehicle, these cameras collect images in real time. Based on the multiple images collected by multiple cameras, the vehicle panoramic system can stitch multiple images into a panoramic image, that is, a 360-degree surround image. The vehicle panoramic system can provide the driver with the surrounding conditions of the vehicle based on the panoramic image and provide visual assistance for the driver's operation.

[0003] Sunlight refers to the light from the sun. Sunlight is white, parallel, and uniform. When stitching multiple images into a panoramic image, if the sunlight is blocked by obstacles, it will cause a sun shadow to appear in the panoramic image (a sun shadow is a shadow cast by an object when the sun is used as a light source to shine on the object), resulting in poor image display and affecting the driver's recognition of the surrounding environment. Summary of the invention

[0004] The present application provides an image processing method, wherein an area around a vehicle is divided into a plurality of grids, and a first grid is any grid among the plurality of grids. The method comprises:

[0005] Determine a first ratio value of the first grid based on the target panoramic image; wherein the first grid corresponds to a first image area in the target panoramic image, and the first ratio value is a ratio value of a shadow area of ​​the first image area to a total area of ​​the first image area;

[0006] For each configured candidate light direction, mapping multiple vertices of the vehicle to multiple ground projection points based on the candidate light direction, wherein the multiple ground projection points constitute multiple shadow surfaces; determining a second ratio value of the first grid based on the multiple shadow surfaces; wherein the first grid includes multiple subgrids, the second ratio value is a ratio value of the number of reference subgrids to the total number of the multiple subgrids, and the reference subgrid is a subgrid having an overlapping area with any shadow surface;

[0007] Based on the difference between the first ratio value and the second ratio value corresponding to each candidate light direction, the candidate light direction corresponding to the minimum difference is selected as the target light direction;

[0008] Brightness compensation is performed on a first image area in the target panoramic image based on the target brightness compensation value corresponding to the target light direction to obtain a compensated first image area.

[0009] The present application provides an image processing device, wherein an area around a vehicle is divided into a plurality of grids, a first grid is any grid among the plurality of grids, and the device comprises:

[0010] a determination module, configured to determine a first ratio value of the first grid based on a target panoramic image; wherein the first grid corresponds to a first image area in the target panoramic image, and the first ratio value is a ratio value of a shadow area of ​​the first image area to a total area of ​​the first image area;

[0011] A processing module, configured to map, for each configured candidate light direction, a plurality of vertices of the vehicle into a plurality of ground projection points based on the candidate light direction, wherein the plurality of ground projection points constitute a plurality of shadow surfaces; and determine a second ratio value of the first grid based on the plurality of shadow surfaces; wherein the first grid includes a plurality of subgrids, the second ratio value is a ratio value of the number of reference subgrids to the total number of the plurality of subgrids, and the reference subgrid is a subgrid having an overlapping area with any shadow surface;

[0012] a selection module, configured to select, based on a difference between the first ratio value and a second ratio value corresponding to each candidate light direction, a candidate light direction corresponding to a minimum difference as a target light direction;

[0013] The compensation module is used to perform brightness compensation on a first image area in the target panoramic image based on a target brightness compensation value corresponding to the target light direction to obtain a compensated first image area.

[0014] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned example image processing method.

[0015] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the image processing method of the above example is implemented.

[0016] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions, and when the machine-executable instructions are executed, the above-mentioned example image processing method is implemented.

[0017] It can be seen from the above technical scheme that in the embodiment of the present application, after obtaining the target panoramic image, the target panoramic image can be brightness compensated to obtain the compensated target panoramic image, thereby eliminating the sun shadow in the target panoramic image, improving the image display effect, and then providing the driver with the vehicle surrounding situation through the target panoramic image, providing visual assistance for the driver's operation. When performing brightness compensation on the target panoramic image, the best light direction can be obtained as the target light direction, and then the target panoramic image is brightness compensated based on the target brightness compensation value corresponding to the target light direction, so that a more accurate target brightness compensation value is used for brightness compensation, the effect of brightness compensation is improved, and the image quality of the target panoramic image is further improved. By dividing the target panoramic image into multiple grids, the best light direction is obtained for each grid as the target light direction, that is, the target brightness compensation value is obtained for each grid, instead of obtaining the target brightness compensation value for each pixel, thereby reducing the calculation amount of the compensation operation, that is, the calculation amount of the compensation operation is very small, saving processor resources, and even a processor with weaker performance can complete the compensation operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of an image processing method in one embodiment of the present application;

[0019] Figure 2 is a flowchart of an image processing method in one embodiment of the present application;

[0020] Figure 3A is a schematic diagram of a fisheye image and a calibrated image in one embodiment of the present application;

[0021] Figure 3B is a schematic diagram of a panoramic image obtained by stitching in one embodiment of the present application;

[0022] Figure 4A and Figure 4B is a schematic diagram of grid cutting of a real scene in one embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a shaded area in one embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a sun shadow in one embodiment of the present application;

[0025] Fig. 7A and Figure 7B is a schematic diagram of rasterization of a sub-grid in one embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a panoramic image with sun shadow removed in one embodiment of the present application;

[0027] Fig. 9A is a schematic structural diagram of an image processing device in one embodiment of the present application;

[0028] Fig. 9B It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0029] In an embodiment of the present application, an image processing method is proposed, which can be applied to an electronic device, and the electronic device can be a vehicle-mounted terminal deployed on a vehicle, or a back-end device, without limitation. The area around the vehicle (such as an area centered on the vehicle) is divided into multiple grids, and the first grid is any grid among the multiple grids. Taking the processing process for the first grid as an example, subsequent processing is performed on each network.

[0030] See also Figure 1 FIG. 1 is a flow chart of the image processing method, which may include:

[0031] Step 101, determining a first ratio value of a first grid based on a target panoramic image; wherein the first grid may correspond to a first image area in the target panoramic image, and the first ratio value may be a ratio value of a shadow area of ​​the first image area to a total area of ​​the first image area.

[0032] Step 102: for each configured candidate light direction, based on the candidate light direction, multiple vertices of the vehicle are mapped to multiple ground projection points, and the multiple ground projection points can form multiple shadow surfaces; based on the multiple shadow surfaces, a second ratio value of the first grid is determined; wherein the first grid may include multiple subgrids, and the second ratio value may be a ratio value of the number of reference subgrids to the total number of the multiple subgrids. For example, the reference subgrid may be a subgrid that has an overlapping area with any shadow surface.

[0033] Step 103: Based on the difference between the first ratio value and the second ratio value corresponding to each candidate light direction, select the candidate light direction corresponding to the minimum difference as the target light direction.

[0034] Step 104 : Perform brightness compensation on the first image area in the target panoramic image based on the target brightness compensation value corresponding to the target light direction to obtain a compensated first image area.

[0035] Exemplarily, the process of determining the area of ​​the shadow region of the first image region may include, but is not limited to: calculating the gradient values ​​of multiple pixel points on the edge of the first image region; based on the gradient value of each pixel point, selecting K pixel points with large gradient values ​​from the multiple pixel points as jumping pixel points, where K may be a positive integer greater than 1; generating at least one shadow region based on the K jumping pixel points; for each shadow region, the edge connecting two jumping pixel points and the edge of the first image region constitute the shadow region; and determining the area of ​​the shadow region of the first image region based on the sum of the areas of all shadow regions.

[0036] Exemplarily, mapping multiple vertices of the vehicle to multiple ground projection points based on the candidate light direction may include, but is not limited to: generating a projection matrix based on the candidate light direction; wherein the candidate light direction may be L=(l x ,l y ,l z ), the projection matrix can be For each vertex, the vertex may be converted into a projection vertex along the candidate light direction based on the projection matrix, and the projection vertex may be mapped to a ground projection point corresponding to the vertex.

[0037] Exemplarily, for each sub-grid of the first grid, if the sub-grid has an overlapping area with any shadow surface, the sub-grid is used as a reference sub-grid. The shadow surface is a triangular shadow surface, and the shadow surface is composed of a first ground projection point, a second ground projection point, and a third ground projection point; if the sum of the first area, the second area, and the third area meets a preset condition, then the sub-grid has an overlapping area with the shadow surface; wherein the first area is an area composed of the center point coordinates of the sub-grid, the first ground projection point, and the second ground projection point, the second area is an area composed of the center point coordinates of the sub-grid, the first ground projection point, and the third ground projection point, and the third area is an area composed of the center point coordinates of the sub-grid, the second ground projection point, and the third ground projection point; wherein the preset conditions include: S PiAB +S PiBC +S PiCA =S i , S PiAB represents the first area, S PiBC Represents the second area, S PiCA represents the third area; where,

[0038]

[0039] Among them, x 1i and 1i is the coordinate of the first ground projection point, x 2i and2i is the coordinate of the second ground projection point, x 3i and 3i are the coordinates of the third ground projection point, and x and y are the coordinates of the center point.

[0040] Exemplarily, brightness compensation is performed on the first image area in the target panoramic image based on the target brightness compensation value corresponding to the target light direction. Before obtaining the compensated first image area, the solar altitude angle is determined based on the target light direction, and the solar altitude angle represents the angle of the center of the sun relative to the horizon; based on the solar altitude angle and a calibrated solar illumination coefficient, the target brightness compensation value corresponding to the first image area is determined.

[0041] Exemplarily, determining the solar altitude angle based on the target light direction may include, but is not limited to: determining the solar altitude angle using the following formula: Among them, θ sun represents the solar altitude angle, (l x ,l y ,l z ) represents the target light direction; l x Indicates the angle value in the x direction of the target light direction, l y Indicates the angle value in the y direction of the target light direction, l z Indicates the angle value in the z direction of the target light direction.

[0042] Exemplarily, based on the solar altitude angle and the calibrated solar illumination coefficient, determining the target brightness compensation value corresponding to the first image area may include but is not limited to: using the following formula to determine the target brightness compensation value: comp =a0+a1θ+a2θ 2 +...+a k θ k ; Among them, I comp Indicates the target brightness compensation value, a0, a1, a2, ..., a k represents the calibrated solar illumination coefficient, k represents the configured value, and θ represents the solar altitude angle.

[0043] Exemplarily, for each solar altitude angle, the calibration process of the solar illumination coefficient at the solar altitude angle may include but is not limited to: obtaining multiple sets of measurement data at the solar altitude angle, wherein for each set of measurement data, the measurement data includes a first sample image with a solar shadow and a second sample image without a solar shadow at the solar altitude angle; calculating the brightness difference between the first sample image and the second sample image; calculating the solar illumination coefficient at the solar altitude angle based on the brightness difference corresponding to each set of measurement data and the solar altitude angle, and calibrating the solar illumination coefficient for the solar altitude angle.

[0044] It can be seen from the above technical scheme that in the embodiment of the present application, after obtaining the target panoramic image, the target panoramic image can be brightness compensated to obtain the compensated target panoramic image, thereby eliminating the sun shadow in the target panoramic image, improving the image display effect, and then providing the driver with the vehicle surrounding situation through the target panoramic image, providing visual assistance for the driver's operation. When performing brightness compensation on the target panoramic image, the best light direction can be obtained as the target light direction, and then the target panoramic image is brightness compensated based on the target brightness compensation value corresponding to the target light direction, so that a more accurate target brightness compensation value is used for brightness compensation, the effect of brightness compensation is improved, and the image quality of the target panoramic image is further improved. By dividing the target panoramic image into multiple grids, the best light direction is obtained for each grid as the target light direction, that is, the target brightness compensation value is obtained for each grid, instead of obtaining the target brightness compensation value for each pixel, thereby reducing the calculation amount of the compensation operation, that is, the calculation amount of the compensation operation is very small, saving processor resources, and even a processor with weaker performance can complete the compensation operation.

[0045] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0046] In the embodiment of the present application, an image processing method is proposed to remove the sun shadow of a multi-channel spliced ​​image (i.e., a target panoramic image). Figure 2 As shown, the image processing method may include:

[0047] Step 201: Acquire a target panoramic image, which may be a panoramic bird's-eye view.

[0048] For example, a plurality of cameras may be installed on the vehicle, such as a camera located in front of the vehicle, a camera located at the rear of the vehicle, a camera located on the left side of the vehicle, and a camera located on the right side of the vehicle. During the driving process of the vehicle, these cameras collect images in real time. Based on the multiple images collected by the multiple cameras, the vehicle-mounted terminal may generate a target panoramic image, which may be a 360-degree surround image. Based on the target panoramic image, the driver may be provided with the surrounding conditions of the vehicle, and visual assistance may be provided for the driver's operation.

[0049] If the image processing method is applied to a vehicle-mounted terminal, after the vehicle-mounted terminal generates a target panoramic image, subsequent steps are performed based on the target panoramic image to complete the image processing method. If the image processing method is applied to a back-end device, after the vehicle-mounted terminal generates a target panoramic image, the target panoramic image is sent to the back-end device, and the back-end device performs subsequent steps based on the target panoramic image to complete the image processing method.

[0050] Exemplarily, the process of “generating a target panoramic image based on multiple images” may include:

[0051] The camera installed on the vehicle can be a fisheye camera. The image collected by the fisheye camera is a distorted fisheye image. The fisheye image does not have the relevant content of position information and depth information. Figure 3A The left side of is a schematic diagram of a fisheye image. Based on this, the fisheye image can also be mapped to a calibrated image, see Figure 3A Shown on the right side is a schematic diagram of the calibrated image of the fisheye image.

[0052] Regarding the process of mapping the fisheye image to the calibrated image, the fisheye image can be mapped to the calibrated image in the world coordinate system based on the camera's internal parameters, the conversion relationship between the image coordinate system and the world coordinate system. There is no restriction on this process, as long as the conversion from the fisheye image to the calibrated image can be completed.

[0053] After performing the above transformations on the fisheye images captured by the camera in front of the vehicle, the fisheye images captured by the camera behind the vehicle, the fisheye images captured by the camera on the left side of the vehicle, and the fisheye images captured by the camera on the right side of the vehicle, multiple calibrated images (i.e., bird's-eye view sub-images) corresponding to the multiple fisheye images can be obtained. Then, the multiple calibrated images are spliced ​​according to their placement in the world coordinate system to obtain a panoramic bird's-eye view, which is a panoramic image in the world coordinate system. This panoramic image is called the initial panoramic image. See Figure 3B As shown, it is a schematic diagram of stitching a panoramic image.

[0054] After obtaining the initial panoramic image, the initial panoramic image can be used as the target panoramic image, or the initial panoramic image can be subjected to image enhancement processing to obtain the target panoramic image. The image enhancement processing can include but is not limited to at least one of the following: grayscale processing, chroma removal processing, and contrast removal processing.

[0055] Step 202: Segment the area around the vehicle into a plurality of grids. The area around the vehicle is the area where the vehicle is located. The area around the vehicle includes the vehicle and may be an area centered on the vehicle. There is no restriction on the area around the vehicle. For example, the area around the vehicle overlaps with the area covered by the target panoramic image, that is, the area covered by the target panoramic image is taken as the area around the vehicle, and the area around the vehicle is segmented into a plurality of grids.

[0056] For example, see Figure 4A As shown, it is a schematic diagram of grid cutting of a real scene, and the area around the vehicle can be divided into multiple grids. Obviously, since the area around the vehicle coincides with the area covered by the target panoramic image, when the area around the vehicle is divided into multiple grids, the target panoramic image can also be divided into multiple grids, and the multiple grids of the target panoramic image correspond to the multiple grids of the area around the vehicle one by one.

[0057] For example, a grid can also be called a raster grid. The width of all grids can be the same, and the height of all grids can be the same, that is, the size of all grids is the same. For each grid, the width of the grid can be the same as the height of the grid, or the width of the grid can be different from the height of the grid. For example, the width and height are the same, and the width of each grid is recorded as l big , let the height of each grid be l big .

[0058] For example, when the area around the vehicle is divided into multiple grids (raster grids), for each grid, the coordinates of the upper left corner vertex of the grid are (i·l big ,j.l big ), the coordinates of the lower left vertex of the grid are ((i+1)·l big ,j.l big ), the coordinates of the upper right corner vertex of the grid are (i·l big ,(j+1)·l big ), the coordinates of the lower right corner vertex of the grid are ((i+1)·l big ,(j+1)·l big ). In addition, the coordinates of the center point of the grid (i.e., the center of the raster rectangle) are

[0059] i is an integer index from -N to N, indicating the horizontal position of the grid, j is an integer index from -M to M, indicating the vertical position of the grid, when i is 0 and j is 0, it indicates the grid where the center of the vehicle is located. -N to N means there are 2N grids in the horizontal direction, -M to M means there are 2M grids in the vertical direction, M and N can be the same, or different.

[0060] Exemplarily, by dividing the target panoramic image into multiple grids, a brightness compensation operation is performed for each grid, rather than for each pixel, thereby reducing the amount of calculation of the compensation operation and saving processor resources. Obviously, after the target panoramic image (the area around the vehicle) is divided into multiple grids, a brightness compensation operation (i.e., a subsequent step) needs to be performed for each grid. For the sake of distinction, the brightness compensation operation of one grid is taken as an example, and the grid is referred to as the first grid. The first grid can be any grid among all the grids, so that the brightness compensation operation is performed on each grid.

[0061] For example, see Figure 4B As shown, it is a schematic diagram of grid cutting of a real scene. One grid can be extracted from all grids as the first grid. The first grid is any grid among all grids.

[0062] Step 203: Determine a first ratio value of a first grid based on a first image area of ​​a target panoramic image, wherein the first grid may correspond to a first image area in the target panoramic image, and the first ratio value may be a ratio value of a shadow area of ​​the first image area to a total area of ​​the first image area.

[0063] Exemplarily, the area around the vehicle overlaps with the area covered by the target panoramic image. When the area around the vehicle is divided into multiple grids, the target panoramic image can also be divided into multiple image areas, and the multiple image areas of the target panoramic image correspond to the multiple grids of the area around the vehicle. Based on this, the image area corresponding to the first grid in the target panoramic image can be determined, and the image area is recorded as the first image area.

[0064] Exemplarily, the process of determining the first scale value of the first grid may include the following steps:

[0065] Step 2031: Calculate the gradient values ​​of multiple pixel points on the edge of the first image area.

[0066] For example, multiple pixels may be selected from the edge of the first image region, and the multiple pixels may be all pixels on the edge of the first image region or some pixels on the edge of the first image region. For each selected pixel, the gradient value of the pixel may be calculated based on the brightness value of the pixel and the brightness values ​​of the pixels surrounding the pixel, and there is no restriction on the calculation method.

[0067] Step 2032: Based on the gradient value of each pixel point, select K pixel points with large gradient values ​​from multiple pixel points as jump pixel points, where K can be a positive integer greater than 1.

[0068] For example, K may be a fixed value, such as 2, 3, etc. Based on this, multiple pixels may be sorted in descending order of gradient values, and the top K pixels may be selected as jump pixels.

[0069] For example, K can be a dynamic value, and a threshold can be pre-configured. For each pixel point, if the gradient value of the pixel point is greater than the threshold, the pixel point is used as a jump pixel point. If the gradient value of the pixel point is not greater than the threshold, the pixel point is not used as a jump pixel point, thereby obtaining K jump pixel points.

[0070] Step 2033: Generate at least one shadow area based on the K jump pixel points. For each shadow area, an edge connecting two jump pixel points and an edge of the first image area constitute the shadow area.

[0071] For example, assuming that the K jumping pixel points include jumping pixel point 1, jumping pixel point 2 and jumping pixel point 3, then jumping pixel point 1 and jumping pixel point 2 are connected to form an edge, and this edge and the edge of the first image area constitute shadow area 1, jumping pixel point 1 and jumping pixel point 3 are connected to form an edge, and this edge and the edge of the first image area constitute shadow area 2, and jumping pixel point 2 and jumping pixel point 3 are connected to form an edge, and this edge and the edge of the first image area constitute shadow area 3.

[0072] For example, see Figure 5 The diagram is a schematic diagram of the shaded area. Figure 5 Taking two jumping pixels as an example, the edge connecting the two jumping pixels and the edge of the first image area form a shadow area Si.

[0073] Step 2034: determine the area of ​​the shadow region of the first image region based on the sum of the areas of all the shadow regions, and determine a first ratio value based on the ratio of the area of ​​the shadow region to the total area of ​​the first image region.

[0074] For example, referring to formula (1), it is an example of calculating the first ratio value.

[0075]

[0076] In formula (1), CountTop (x,y) Represents the first scale value of the first grid (x, y), Si represents the area of ​​the i-th shadow area, and can represent the total number of pixels in the i-th shadow area. The value range of i is 1-n, that is, there are n shadow areas in total. represents the sum of the areas of all shadow regions (ie, n shadow regions), that is, the area of ​​the shadow regions of the first image region. represents the total area of ​​the first image region, l big represents the width of the first image area, l big It also represents the height of the first image area.

[0077] At this point, step 203 is completed, and the first ratio value CountTop of the first grid can be obtained. (x,y) , the first ratio value of the first grid can also be called the first weight value of the first grid. The above method of calculating the first ratio value can also be called the sun shadow separation matching method in the real scene.

[0078] Step 204 : for each configured candidate light direction, multiple vertices of the vehicle are mapped to multiple ground projection points based on the candidate light direction, and the multiple ground projection points can form multiple shadow surfaces.

[0079] Exemplarily, the process of mapping multiple vertices to multiple ground projection points may include the following steps:

[0080] Step 2041: Model the vehicle to obtain multiple vertices of the vehicle.

[0081] For example, if the structural model of a vehicle is known, 3D modeling technology can be used to build a digital model of the vehicle, and there is no restriction on this modeling process. For example, in the digital model of a 3D modeled vehicle, the digital model of the vehicle is composed of vertices and faces, the vertices define the spatial position, the edges connect the vertices to form polygons, and the faces are polygons composed of vertices and edges. The faces in this embodiment can be composed of triangles or other polygons. The following takes the faces composed of triangles as an example.

[0082] For each vertex of the digital model of the vehicle (i.e. each vertex of the vehicle), the vertex can be represented by the following matrix: v i Represents the coordinates of the vertex, x i Indicates the x-coordinate of the vertex, y i Indicates the y-coordinate of the vertex, z i Indicates the z-coordinate of the vertex.

[0083] Taking the triangular surface as an example, each surface of the digital model of the vehicle can be represented by three vertices, that is, each surface of the digital model of the vehicle is represented as follows: j =(v j1 ,v j2 ,v j3 ).

[0084] Step 2042: Obtain multiple configured candidate light directions.

[0085] For example, different light directions will produce different shadows on the vehicle, see Figure 6 As shown, it is a schematic diagram of the sun shadow produced when the light direction of each angle illuminates the vehicle. Therefore, multiple light directions can be pre-configured, and these light directions are called candidate light directions. In the subsequent process, it is necessary to select an optimal light direction from multiple candidate light directions as the target light direction. Regarding how to configure multiple light directions, there is no limitation in this embodiment, and it can be configured based on experience.

[0086] Step 2043: For each candidate light direction, generate a projection matrix based on the candidate light direction.

[0087] For example, see formula (2), which is an example of the projection matrix.

[0088]

[0089] In formula (2), M represents the projection matrix, which can also be called M directional , L=(l x ,l y ,l z ) represents the candidate light direction, l x Indicates the angle value in the x direction of the candidate light direction, l y Indicates the angle value in the y direction of the candidate light direction, l z Indicates the angle value in the z direction of the candidate light direction.

[0090] Step 2044: for each vertex of the vehicle, convert the vertex into a projection vertex along the candidate light direction based on the projection matrix, and the projection vertex along the candidate light direction is the coordinate after projection.

[0091] For example, see formula (3), which is an example of transformation based on the projection matrix.

[0092]

[0093] In formula (3), v i represents the coordinates of the vertex, v′ i Represents the coordinates of the projection vertex along the direction of the candidate light, that is, the projection matrix M is used to project the vertex v i Transform to get the projection vertex v′ i .

[0094] Step 2045: Map the projection vertex to the ground projection point corresponding to the vertex.

[0095] For example, referring to formula (4), it is an example of mapping the projection vertex to the ground projection point.

[0096]

[0097] In formula (4), g i Represents the vertex v i The corresponding ground projection point, v′ i Represents the projected vertex, that is, the projected vertex v′ i The coordinate in the z direction is set to 0, and the ground projection point g is obtained. i .

[0098] At this point, step 204 is completed. For each candidate light direction, multiple vertices of the vehicle can be mapped to multiple ground projection points g based on the candidate light direction. i , ground projection point g i is the coordinate point on the ground, thus obtaining a set of ground projection points g i, i = 1, 2, 3, ..., n, n represents the total number of ground projection points.

[0099] Obviously, these ground projection points can form multiple shadow surfaces. Taking the shadow surface as a triangular shadow surface as an example, every three ground projection points form a shadow surface. For example, shadow surface Fg1 = (g1, g2, g3), shadow surface Fg2 = (g4, g5, g6), ..., shadow surface

[0100] For example, when modeling a vehicle, assuming that vertex 1, vertex 2, and vertex 3 form a triangular surface, then the ground projection point corresponding to vertex 1, the ground projection point corresponding to vertex 2, and the ground projection point corresponding to vertex 3 can form a triangular shadow surface.

[0101] Step 205: Determine a second ratio value of the first grid based on the multiple shadow surfaces. The first grid may include multiple sub-grids, and the second ratio value may be a ratio value of the number of reference sub-grids to the total number of the multiple sub-grids. For example, the reference sub-grid may be a sub-grid that has an overlapping area with any shadow surface.

[0102] Exemplarily, the first grid may be divided into a plurality of subgrids, and for each subgrid of the first grid, if the subgrid has an overlapping area with any shadow surface, the subgrid is used as a reference subgrid. If the subgrid does not have an overlapping area with any shadow surface, the subgrid is not used as a reference subgrid.

[0103] The number of reference sub-grids in the first grid may be counted, and a ratio of the number of reference sub-grids to the total number of all sub-grids in the first grid may be used as a second ratio value of the first grid.

[0104] Exemplarily, the process of determining the second ratio value of the first grid may include the following steps:

[0105] Step 2051: The first grid may be divided into a plurality of sub-grids, which may also be referred to as grating grids.

[0106] For example, the width of all subgrids can be the same, and the height of all subgrids can be the same, that is, the size of all subgrids is the same. For each subgrid, the width of the subgrid can be the same as the height of the subgrid, or the width of the subgrid can be different from the height of the subgrid. For example, if the width and height are the same, the width of each subgrid is recorded as l min , let the height of each subgrid be l min .

[0107] For example, when the first grid is divided into a plurality of sub-grids, for each sub-grid, the coordinates of the upper left corner vertex of the sub-grid may be (i·lmin ,j.l min ), the coordinates of the lower left corner vertex of the sub-grid are ((i+1)·l min ,j.l min ), the coordinates of the upper right corner vertex of the sub-grid are (i·l min ,(j+1)·l min ), the coordinates of the lower right corner vertex of the sub-grid are ((i+1)·l min ,(j+1)·l min ). In addition, the coordinates of the center point of the sub-grid (i.e., the center of the raster rectangle) are

[0108] i is an integer index from -A to A, indicating the position of the subgrid in the horizontal direction, j is an integer index from -B to B, indicating the position of the subgrid in the vertical direction, and when i is 0 and j is 0, it indicates the subgrid where the center of the vehicle is located. -A to A means there are 2A grids in the horizontal direction, -B to B means there are 2B grids in the vertical direction, B and A can be the same, or B and A can be different.

[0109] It should be noted that the coordinates of the sub-grid here are the coordinates within the range of the area surrounding the vehicle (the target panoramic image), rather than the coordinates within the range of a certain grid (ie, the first grid).

[0110] For example, see Fig. 7A As shown in FIG. 1 , it is a schematic diagram of the rasterization of subgrids. In the world coordinate system, all grids (i.e., all first grids) can be cut into a large number of subgrids, and the side length of each subgrid is l min , the side length and height of each sub-grid is l min , the coordinates of each sub-grid can be found in the above example.

[0111] Step 2052: for each sub-grid of the first grid, determine whether the sub-grid has an overlapping area with the shadow surface. If the sub-grid has an overlapping area with any shadow surface, the sub-grid is used as a reference sub-grid. If the sub-grid does not have an overlapping area with any shadow surface, the sub-grid is not used as a reference sub-grid.

[0112] Exemplarily, after mapping multiple vertices to multiple ground projection points, multiple shadow surfaces can be obtained, and each shadow surface is a triangular shadow surface. For example, a triangular shadow surface can be composed of a first ground projection point, a second ground projection point, and a third ground projection point. On this basis, for each sub-grid of the first grid, if the sum of the first area, the second area, and the third area corresponding to the sub-grid meets the preset conditions, then the sub-grid and the shadow surface may have an overlapping area, otherwise, the sub-grid and the shadow surface may not have an overlapping area. Among them, the first area can be the area composed of the center point coordinates of the sub-grid, the first ground projection point, and the second ground projection point, the second area can be the area composed of the center point coordinates of the sub-grid, the first ground projection point, and the third ground projection point, and the third area can be the area composed of the center point coordinates of the sub-grid, the second ground projection point, and the third ground projection point.

[0113] For example, formula (5) can be used to determine whether there is an overlapping area between the sub-grid and the shadow surface:

[0114]

[0115] In formula (5), if S PiAB +S PiBC +S PiCA =S i , it means that the sum of the first area, the second area and the third area meets the preset conditions, f i (x, y) is 1, which means there is an overlap between the subgrid (x, y) and the i-th shadow surface (i.e., the i-th shadow surface among all shadow surfaces). Otherwise, if S PiAB +S PiBC +S PiCA Not equal to S i , it means that the sum of the first area, the second area and the third area does not meet the preset conditions, f i (x, y) is 0, which means there is no overlapping area between the sub-grid (x, y) and the i-th shadow surface. As can be seen from the above, the preset conditions may include but are not limited to: S PiAB +S PiBC +S PiCA =S i , there is no restriction on this preset condition.

[0116] In formula (5), S PiAB represents the first area, and S PiBC represents the second area, and S PiCA represents the third area, and

[0117] For example, x1i and 1i is the coordinate of the first ground projection point, x 2i and 2i is the coordinate of the second ground projection point, x 3i and 3i are the coordinates of the third ground projection point, see step 204, the triangular shadow surface is composed of three ground projection points, and the coordinates of these three ground projection points are known. Therefore, for each triangular shadow surface, the coordinates of the three ground projection points of the triangular shadow surface can be obtained.

[0118] For example, x and y are the coordinates of the center point of the sub-grid of the first grid. Referring to step 2051, when the first grid is divided into a plurality of sub-grids, the coordinates of the center point of each sub-grid can be obtained.

[0119] To summarize, for each sub-mesh, each triangular shadow surface can be traversed in turn. If, based on the coordinates of the center point of the sub-mesh and the coordinates of the three ground projection points of the triangular shadow surface, the sum of the first area, the second area and the third area meets the preset conditions, then the sub-mesh has an overlapping area with the triangular shadow surface. Otherwise, the sub-mesh does not have an overlapping area with the triangular shadow surface.

[0120] For example, since multiple ground projection points can form multiple triangular shadow surfaces, it is also possible to determine whether there is an overlapping area between the sub-mesh and all triangular shadow surfaces. For example, formula (6) can be used to determine whether there is an overlapping area between the sub-mesh and all triangular shadow surfaces:

[0121]

[0122] In formula (6), if there is an overlapping area between the sub-mesh (x, y) and the i-th triangle shadow surface, then f i (x, y) is 1. If there is no overlapping area between the sub-mesh (x, y) and the i-th triangle shadow surface, then f i (x, y) is 0, and the value range of i is 1-n, that is, there are n triangular shadow surfaces in total. Obviously, if there is an overlapping area between the sub-mesh (x, y) and at least one triangular shadow surface, then F(x, y) is 1, indicating that there are overlapping areas between the sub-mesh and all triangular shadow surfaces. In other words, as long as there is an overlapping area between the sub-mesh (x, y) and at least one triangular shadow surface, it means that there are overlapping areas between the sub-mesh and all triangular shadow surfaces. Alternatively, if there is no overlapping area between the sub-mesh (x, y) and all triangular shadow surfaces, then F(x, y) is 0, indicating that there is no overlapping area between the sub-mesh and all triangular shadow surfaces.

[0123] From the above, it can be seen that for each sub-mesh of the first mesh, if the sub-mesh has an overlapping area with all the triangle shadow surfaces, the sub-mesh is used as a reference sub-mesh, that is, the sub-mesh with F(x, y) being 1 is used as a reference sub-mesh. If the sub-mesh does not have an overlapping area with all the triangle shadow surfaces, the sub-mesh is not used as a reference sub-mesh, that is, the sub-mesh with F(x, y) being 0 is not used as a reference sub-mesh.

[0124] For example, see Figure 7B As shown in FIG. 1 , it is a schematic diagram of rasterization of subgrids. All first grids can be cut into a large number of subgrids, and the projected surface is rasterized to obtain shadow area pixels, that is, triangular shadow surfaces. Figure 7B The black area in the figure represents the triangle shadow surface, that is, the triangle shadow surface is filled with black. Figure 7B In the example, the black area may include multiple triangular shadow surfaces.

[0125] Step 2053: Count the number of reference sub-grids in the first grid.

[0126] For example, in step 2052, it may be determined whether each sub-grid in the first grid belongs to a reference sub-grid or does not belong to a sub-grid, so that the number of reference sub-grids may be counted.

[0127] Step 2054: Determine a second ratio value of the first grid based on the ratio value of the number of reference subgrids to the total number of all subgrids in the first grid. For example, the second ratio value of the first grid can be determined using the following formula: Count (x,y) =P1 / P2, in the above formula, Count (x,y) represents the second proportional value of the first grid, P1 represents the number of reference sub-grids, P2 represents the total number of all sub-grids in the first grid, and the second proportional value of the first grid can also be called the second weight value of the first grid.

[0128] At this point, step 205 is completed, and the second ratio value Count of the first grid can be obtained. (x,y) .

[0129] Step 206: Based on the difference between the first ratio value and the second ratio value corresponding to each candidate light direction, select the candidate light direction corresponding to the minimum difference as the target light direction.

[0130] For example, multiple candidate light directions may be pre-configured, and for each candidate light direction, the second scale value of the first grid is determined through steps 204 and 205, and this second scale value is the second scale value corresponding to the candidate light direction, thereby obtaining the second scale value corresponding to each candidate light direction.

[0131] Calculate the difference between the first ratio value and the second ratio value. For example, use the following formula to calculate the difference between the first ratio value and the second ratio value: For example, CountTop (x,y) It can represent the first scale value of the first grid (x, y), Count (x,y) It can represent the second scale value of the first grid (x, y) corresponding to the i-th candidate light direction. i It can represent the difference between the first ratio value and the second ratio value corresponding to the i-th candidate light direction. It can be seen from the above formula that the minimum value of the difference between the first ratio value and the second ratio value of all first networks can be traversed to obtain the best light direction of the first grid (x, y).

[0132] For example, based on the difference between the first ratio value and the second ratio value corresponding to each candidate light direction, a minimum difference value, Diff min =min(Diff1,Diff2,...,Diff n ), Diff min represents the minimum difference, Diff1 represents the difference between the first ratio value and the second ratio value corresponding to the first candidate light direction, ..., Diff n Represents the difference between the first ratio value and the second ratio value corresponding to the nth candidate light direction. After obtaining the minimum difference, the candidate light direction corresponding to the minimum difference can be selected as the target light direction. The target light direction is the best light direction for the first grid (x, y), that is, the difference between the second ratio value corresponding to the target light direction and the first ratio value is the smallest. For example, the above comparison process can be called the pattern matching process of the sun shadow. The first ratio value and the second ratio value corresponding to each candidate light direction can be compared to calculate the minimum Diff min The optimal light direction is used as the target light direction.

[0133] Step 207: Determine the solar altitude angle based on the target light direction. For example, the solar altitude angle may be determined using formula (7). Of course, formula (7) is only an example and is not limited thereto.

[0134]

[0135] In formula (7), θ sun The solar altitude angle represents the angle of the sun's center relative to the horizon. x ,l y ,l z ) represents the target light direction, which represents the direction of the sun’s light, i.e. the sun’s optimal light direction. xIndicates the angle value in the x direction of the target light direction, l y Indicates the angle value in the y direction of the target light direction, l z Indicates the angle value in the z direction of the target light direction.

[0136] Step 208: Determine a target brightness compensation value corresponding to the first image area based on the solar altitude angle and the calibrated solar illumination coefficient, where the target brightness compensation value is used to perform brightness compensation on the first image area.

[0137] Exemplarily, after obtaining the target light direction (optimal light direction), the solar altitude angle can be obtained through the target light direction, and the brightness of the object's solar shadow can be compensated by the solar altitude angle. The solar altitude angle is the angle of the center of the sun relative to the horizon. When the solar altitude angle is low, the incident angle of sunlight is large, resulting in a longer and shallower shadow. When the solar altitude angle is high, the sunlight is almost vertically irradiated, and the shadow is shorter and deeper. Under sunlight, the brightness of an object depends on the superposition of the direct diffuse reflection of sunlight and the diffuse reflection of ambient light, while the object in the shadow only has the diffuse reflection of ambient light.

[0138] Based on the above principle, in this embodiment, a function related to the sun altitude angle can be designed to represent the brightness compensation value I of the shadow comp , that is, the brightness compensation value I comp is a function related to the solar altitude angle. Based on this, the target brightness compensation value corresponding to the first image area can be determined using the following formula (8):

[0139] I comp =f(θ)=a0+a1θ+a2θ 2 +...+a k θ k Formula (8)

[0140] In formula (8), I comp =f(θ) represents the brightness compensation value I comp is a function related to the solar altitude angle θ, which may be f(θ). In this embodiment, there is no restriction on this function f(θ), which may be set according to experience. 2 +...+a k θ k For example.

[0141] In formula (8), I comp represents the target brightness compensation value corresponding to the first image area, a0, a1, a2, ..., a krepresents the calibrated solar illumination coefficient, k represents the configured value, such as 3, 4, 5, etc., and θ represents the solar altitude angle. Obviously, the solar illumination coefficient is a known value. Therefore, after substituting the solar altitude angle θ into formula (8), the target brightness compensation value I corresponding to the first image area can be obtained. comp .

[0142] In a possible implementation, for each solar altitude angle, it is necessary to pre-calibrate the solar illumination coefficient at the solar altitude angle. The calibration process of the solar illumination coefficient at the solar altitude angle may include:

[0143] Step S11, for each solar altitude angle, multiple groups of measurement data under the solar altitude angle are obtained, and for each group of measurement data, the measurement data includes a first sample image with solar shadow under the solar altitude angle and a second sample image without solar shadow under the solar altitude angle.

[0144] For example, for the solar altitude angle A, when the angle of the sun center relative to the horizon is this angle A, the panoramic image is made to have a sun shadow by moving the vehicle, and this panoramic image is called the first sample image. Then, the panoramic image is made to have no sun shadow by moving the vehicle, and this panoramic image is called the second sample image. The first sample image and the second sample image form a set of measurement data.

[0145] By repeating the above operation, multiple sets of measurement data at the solar altitude angle A can be obtained.

[0146] Step S12: For each set of measurement data, calculate the brightness difference between the first sample image in the measurement data and the second sample image in the measurement data, and the brightness difference can be used as a brightness compensation value. For example, calculate the first brightness average value of the first sample image, calculate the second brightness average value of the second sample image, and the brightness difference is the absolute value of the difference between the first brightness average value and the second brightness average value.

[0147] Step S13: Calculate the solar illumination coefficient at the solar altitude angle based on the brightness difference corresponding to each set of measurement data and the solar altitude angle, and calibrate the solar illumination coefficient for the solar altitude angle.

[0148] For example, based on the brightness difference corresponding to each set of measurement data and the solar altitude angle, the target value to be optimized can be calculated. See formula (9), which is an example of calculating the target value to be optimized.

[0149]

[0150] In formula (9), S represents the target value to be optimized, I compiIt indicates the brightness difference corresponding to the i-th set of measurement data, which is a known value. The value range of i is 1-n, that is, there are a total of n sets of measurement data corresponding to the brightness difference. f(θ)=a0+a1θ+a2θ 2 +...+a k θ k , θ represents the solar altitude angle, such as the solar altitude angle A, which is a known value. a0, a1, a2, ..., a k is the solar illumination coefficient to be calibrated and is an unknown value.

[0151] In formula (9), the target value S to be optimized and the solar illumination coefficient are unknown values, and the target value S to be optimized and the solar illumination coefficient are associated. By trying different solar illumination coefficients, the target value S to be optimized is minimized. The solar illumination coefficient under the minimum target value S to be optimized (referred to as the solar illumination coefficient X1) is the solar illumination coefficient under the solar altitude angle, and the solar illumination coefficient X1 is calibrated for the solar altitude angle.

[0152] For example, in the process of trying different solar illumination coefficients to minimize the target value S to be optimized, the least square method can be used to calculate the solar illumination coefficient. For example, the above solution process can be converted into a linear equation about the solar illumination coefficient, which can be expressed in the following matrix form: Aa=b, where A is a matrix composed of the powers of the solar altitude angle θ, a is the coefficient vector of the solar illumination coefficient, and b is a matrix composed of I compi The vector obtained by multiplying the value of by the power of θ and summing them. On this basis, the least square method can be used to calculate the solar illumination coefficient. In this embodiment, the calculation process of the solar illumination coefficient is not limited.

[0153] In summary, for each solar altitude angle, the solar illumination coefficient under the solar altitude angle can be calculated by using steps S11 to S13, and the solar illumination coefficient is calibrated for the solar altitude angle. See Table 1, which shows the mapping relationship between the solar altitude angle and the solar illumination coefficient corresponding to the solar altitude angle.

[0154] Table 1

[0155] Sun Altitude Angle Solar radiation factor Sun altitude angle A Sunlight coefficient X1 Sun altitude angle B Sunlight coefficient X2 … …

[0156] In step 208, after determining the solar altitude angle based on the target light direction, the solar altitude angle can be used to query Table 1 to obtain the solar illumination coefficient corresponding to the solar altitude angle (i.e., the calibrated solar illumination coefficient). In this way, the target brightness compensation value can be determined based on the solar altitude angle and the solar illumination coefficient.

[0157] Step 209 : Perform brightness compensation on the first image area in the target panoramic image based on the target brightness compensation value corresponding to the first image area to obtain a compensated first image area.

[0158] For example, when performing brightness compensation on the first image area, it can be determined whether there is a sun shadow in the first image area. If not, brightness compensation is not performed on the first image area. If so, the brightness value of the sun shadow area of ​​the first image area is added to the target brightness compensation value to obtain the compensated brightness value.

[0159] Obviously, when the first grid corresponds to each image area of ​​the target panoramic image in sequence, brightness compensation can be performed on each image area of ​​the target panoramic image in sequence, so as to obtain a compensated target panoramic image, thus eliminating the sun shadow in the target panoramic image and improving the image display effect. Figure 8 FIG. 1 is a schematic diagram of a panoramic image with sun shadow removed, and the image display effect is significantly improved.

[0160] In a possible implementation manner, all data involved in this embodiment (such as the target panoramic image, etc.) are acquired and used only with the knowledge and authorization of the relevant users.

[0161] It can be seen from the above technical solution that in the embodiment of the present application, when brightness compensation is performed on the target panoramic image, the best light direction can be obtained as the target light direction, and then the target panoramic image can be brightness compensated based on the target brightness compensation value corresponding to the target light direction, so that a more accurate target brightness compensation value is used for brightness compensation, the effect of brightness compensation is improved, and the image quality of the target panoramic image is further improved. By dividing the target panoramic image into multiple grids, the best light direction is obtained for each grid as the target light direction, that is, the target brightness compensation value is obtained for each grid, instead of obtaining the target brightness compensation value for each pixel, thereby reducing the amount of calculation of the compensation operation, that is, the amount of calculation of the compensation operation is very small, saving processor resources, and even a processor with weaker performance can complete the compensation operation.

[0162] Based on the same application concept as the above method, an image processing device is proposed in an embodiment of the present application, wherein the area around the vehicle is divided into a plurality of grids, and the first grid is any grid among the plurality of grids, see Fig. 9A FIG. 1 is a schematic diagram of the structure of the image processing device, and the device may include:

[0163] A determination module 911 is configured to determine a first ratio value of the first grid based on a target panoramic image; wherein the first grid corresponds to a first image area in the target panoramic image, and the first ratio value is a ratio value of a shadow area of ​​the first image area to a total area of ​​the first image area;

[0164] The processing module 912 is configured to map multiple vertices of the vehicle into multiple ground projection points based on each configured candidate light direction, wherein the multiple ground projection points constitute multiple shadow surfaces; determine a second ratio value of the first grid based on the multiple shadow surfaces; wherein the first grid includes multiple sub-grids, the second ratio value is a ratio value of the number of reference sub-grids to the total number of the multiple sub-grids, and the reference sub-grid is a sub-grid having an overlapping area with any shadow surface;

[0165] A selection module 913 is used to select the candidate light direction corresponding to the minimum difference as the target light direction based on the difference between the first ratio value and the second ratio value corresponding to each candidate light direction;

[0166] The compensation module 914 is configured to perform brightness compensation on the first image area in the target panoramic image based on the target brightness compensation value corresponding to the target light direction to obtain a compensated first image area.

[0167] Exemplarily, when determining the area of ​​the shadow area of ​​the first image area, the determination module 911 is specifically used to: calculate the gradient values ​​of multiple pixel points on the edge of the first image area; based on the gradient value of each pixel point, select K pixel points with large gradient values ​​from the multiple pixel points as jumping pixel points, where K is a positive integer greater than 1; generate at least one shadow area based on the K jumping pixel points; for each shadow area, the edge connecting two jumping pixel points and the edge of the first image area constitute the shadow area; and determine the area of ​​the shadow area of ​​the first image area based on the sum of the areas of all shadow areas.

[0168] Exemplarily, when the processing module 912 maps multiple vertices of the vehicle to multiple ground projection points based on the candidate light direction, it is specifically used to: generate a projection matrix based on the candidate light direction; for each vertex, convert the vertex into a projection vertex along the candidate light direction based on the projection matrix, and map the projection vertex to the ground projection point corresponding to the vertex.

[0169] Exemplarily, for each sub-grid of the first grid, if the sub-grid has an overlapping area with any shadow surface, the sub-grid is used as a reference sub-grid; wherein the shadow surface is a triangular shadow surface, and the shadow surface is composed of a first ground projection point, a second ground projection point, and a third ground projection point; if the sum of the first area, the second area, and the third area meets a preset condition, then the sub-grid has an overlapping area with the shadow surface; wherein the first area is an area composed of the center point coordinates of the sub-grid, the first ground projection point, and the second ground projection point, the second area is an area composed of the center point coordinates, the first ground projection point, and the third ground projection point, and the third area is an area composed of the center point coordinates, the second ground projection point, and the third ground projection point; wherein the preset condition includes: S PiAB +S PiBC +S PiCA =S i , S PiAB represents the first area, S PiBC represents the second area, S PiCA represents the third area; wherein,

[0170] Among them, x 1i and 1i is the coordinate of the first ground projection point, x 2i and 2i is the coordinate of the second ground projection point, x 3i and 3i are the coordinates of the third ground projection point, and x and y are the coordinates of the center point.

[0171] Exemplarily, the compensation module 914 is also used to determine the solar altitude angle based on the target light direction, where the solar altitude angle represents the angle of the center of the sun relative to the horizon; based on the solar altitude angle and a calibrated solar illumination coefficient, determine the target brightness compensation value corresponding to the first image area.

[0172] Exemplarily, when the compensation module 914 determines the solar altitude angle based on the target light direction, it is specifically used to: determine the solar altitude angle using the following formula: Among them, θ sun represents the solar altitude angle, (l x ,l y ,l z ) represents the target light direction; wherein, l x Indicates the angle value of the x direction in the target light direction, l y Indicates the angle value in the y direction of the target light direction, l zIndicates the angle value in the z direction of the target light direction.

[0173] Exemplarily, the compensation module 914 determines the target brightness compensation value corresponding to the first image area based on the solar altitude angle and the calibrated solar illumination coefficient by using the following formula to determine the target brightness compensation value: comp =a0+a1θ+a2θ 2 +...+a k θ k ;

[0174] Among them, I comp represents the target brightness compensation value, a0, a1, a2, ..., a k represents the calibrated solar illumination coefficient, k represents the configured value, and θ represents the solar altitude angle.

[0175] Exemplarily, for each solar altitude angle, for the calibration process of the solar illumination coefficient at the solar altitude angle, the compensation module 914 is also used to obtain multiple groups of measurement data at the solar altitude angle, and for each group of measurement data, the measurement data includes a first sample image with a solar shadow and a second sample image without a solar shadow at the solar altitude angle; calculate the brightness difference between the first sample image and the second sample image; based on the brightness difference corresponding to each group of measurement data and the solar altitude angle, calculate the solar illumination coefficient at the solar altitude angle, and calibrate the solar illumination coefficient for the solar altitude angle.

[0176] Based on the same application concept as the above method, an electronic device is proposed in the embodiment of the present application, see Fig. 9B As shown, the electronic device includes: a processor 921 and a machine-readable storage medium 922, the machine-readable storage medium 922 stores machine-executable instructions that can be executed by the processor 921; the processor 921 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of this application.

[0177] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0178] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0179] Based on the same application concept as the above method, an embodiment of the present application also provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it can implement the image processing method disclosed in the above example of the present application.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0181] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: The area around the vehicle is divided into a plurality of grids, the first grid is any grid among the plurality of grids, and the method comprises: Determine a first ratio value of the first grid based on the target panoramic image; wherein the first grid corresponds to a first image area in the target panoramic image, and the first ratio value is a ratio value of a shadow area of ​​the first image area to a total area of ​​the first image area; For each configured candidate light direction, mapping multiple vertices of the vehicle to multiple ground projection points based on the candidate light direction, wherein the multiple ground projection points constitute multiple shadow surfaces; determining a second ratio value of the first grid based on the multiple shadow surfaces; wherein the first grid includes multiple subgrids, the second ratio value is a ratio value of the number of reference subgrids to the total number of the multiple subgrids, and the reference subgrid is a subgrid having an overlapping area with any shadow surface; Based on the difference between the first ratio value and the second ratio value corresponding to each candidate light direction, the candidate light direction corresponding to the minimum difference is selected as the target light direction; Brightness compensation is performed on a first image area in the target panoramic image based on the target brightness compensation value corresponding to the target light direction to obtain a compensated first image area.

2. The method according to claim 1, characterized in that The process of determining the area of ​​the shadow region of the first image region includes: Calculating gradient values ​​of a plurality of pixel points on the edge of the first image area; Based on the gradient value of each pixel point, K pixel points with large gradient values ​​are selected from the multiple pixel points as jump pixel points, where K is a positive integer greater than 1; Generate at least one shadow area based on the K jumping pixels; for each shadow area, an edge connecting two jumping pixels and an edge of the first image area constitute the shadow area; The area of ​​the shadow region of the first image region is determined based on the sum of the areas of all the shadow regions.

3. The method according to claim 1, characterized in that The mapping of a plurality of vertices of the vehicle to a plurality of ground projection points based on the candidate light direction comprises: Generate a projection matrix based on the candidate light direction; wherein the candidate light direction is L=(l x ,l y ,l z ), the projection matrix is For each vertex, the vertex is converted into a projection vertex along the candidate light direction based on the projection matrix, and the projection vertex is mapped to a ground projection point corresponding to the vertex.

4. The method according to claim 1, characterized in that For each sub-grid of the first grid, if the sub-grid has an overlapping area with any shadow surface, the sub-grid is used as a reference sub-grid; The shadow surface is a triangular shadow surface, and the shadow surface is composed of a first ground projection point, a second ground projection point, and a third ground projection point; if the sum of the first area, the second area, and the third area meets a preset condition, then there is an overlapping area between the sub-grid and the shadow surface; wherein the first area is an area composed of the center point coordinates of the sub-grid, the first ground projection point, and the second ground projection point, the second area is an area composed of the center point coordinates, the first ground projection point, and the third ground projection point, and the third area is an area composed of the center point coordinates, the second ground projection point, and the third ground projection point; The preset conditions include: PiAB +S PiBC +S PiCA =S i , S PiAB represents the first area, S PiBC represents the second area, S PiCA represents the third area; wherein, Among them, x 1i and 1i is the coordinate of the first ground projection point, x 2i and 2i is the coordinate of the second ground projection point, x 3i and 3i are the coordinates of the third ground projection point, and x and y are the coordinates of the center point.

5. The method according to claim 1, characterized in that Before performing brightness compensation on the first image area in the target panoramic image based on the target brightness compensation value corresponding to the target light direction to obtain the compensated first image area, the method further includes: The solar altitude angle is determined based on the target light direction, wherein the solar altitude angle represents the angle of the center of the sun relative to the horizon; and the target brightness compensation value corresponding to the first image area is determined based on the solar altitude angle and a calibrated solar illumination coefficient.

6. The method according to claim 5, characterized in that The determining of the solar altitude angle based on the target light direction comprises: The solar altitude angle is determined using the following formula: Among them, θ sun represents the solar altitude angle, (l x ,l y ,l z ) represents the target light direction; Among them, l x Indicates the angle value of the x direction in the target light direction, l y Indicates the angle value in the y direction of the target light direction, l z Indicates the angle value in the z direction of the target light direction.

7. The method according to claim 5, characterized in that Determining the target brightness compensation value corresponding to the first image area based on the sun altitude angle and the calibrated sun illumination coefficient includes: The target brightness compensation value is determined by the following formula: comp =a0+a1θ+a2θ 2 +...+a k θ k ; Among them, I comp represents the target brightness compensation value, a0, a1, a2, ..., a k represents the calibrated solar illumination coefficient, k represents the configured value, and θ represents the solar altitude angle.

8. The method according to claim 5 or 7, characterized in that: For each solar altitude angle, the calibration process of the solar illumination coefficient at the solar altitude angle includes: Acquire multiple groups of measurement data at the solar altitude angle, wherein for each group of measurement data, the measurement data includes a first sample image with a solar shadow and a second sample image without a solar shadow at the solar altitude angle; calculate a brightness difference between the first sample image and the second sample image; Based on the brightness difference corresponding to each set of measurement data and the solar altitude angle, the solar illumination coefficient at the solar altitude angle is calculated, and the solar illumination coefficient is calibrated for the solar altitude angle.

9. An image processing device, characterized in that: The area around the vehicle is divided into a plurality of grids, the first grid is any grid among the plurality of grids, and the device comprises: a determination module, configured to determine a first ratio value of the first grid based on a target panoramic image; wherein the first grid corresponds to a first image area in the target panoramic image, and the first ratio value is a ratio value of a shadow area of ​​the first image area to a total area of ​​the first image area; A processing module, configured to map, for each configured candidate light direction, a plurality of vertices of the vehicle into a plurality of ground projection points based on the candidate light direction, wherein the plurality of ground projection points constitute a plurality of shadow surfaces; and determine a second ratio value of the first grid based on the plurality of shadow surfaces; wherein the first grid includes a plurality of subgrids, the second ratio value is a ratio value of the number of reference subgrids to the total number of the plurality of subgrids, and the reference subgrid is a subgrid having an overlapping area with any shadow surface; a selection module, configured to select, based on a difference between the first ratio value and a second ratio value corresponding to each candidate light direction, a candidate light direction corresponding to a minimum difference as a target light direction; The compensation module is used to perform brightness compensation on a first image area in the target panoramic image based on a target brightness compensation value corresponding to the target light direction to obtain a compensated first image area.

10. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-8.

Images