Image processing method, device, equipment and medium

Abstract

The present invention discloses an image processing method, device, equipment and medium, and relates to the field of image processing technology. The gain value of each YUV channel of the second visual sensor is determined according to the mean value of each channel on the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor, and then the first area is directly colored by the RGB color obtained by the first visual sensor sampling the texture of the first area; when the second area is colored, the RGB value obtained after the second visual sensor samples the texture of the second area is converted into a YUV value, and the gain value of each YUV channel of the second visual sensor determined before is used for calibration, so as to determine the final RGB color of the second area, and the second area is colored by the final RGB color. It can be seen that in this method, the gain value is applied to the coloring process, so as to achieve balanced processing of brightness and color, and improve the visual experience of the user.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, equipment and medium. Background Art

[0002] The vehicle-mounted surround view system obtains input images from multiple surround view lenses and finally stitches the multiple images together. Due to the influence of the parameters of each lens and the different environmental locations, the stitched images will have certain differences in brightness and color at the stitching point, affecting the user's visual experience.

[0003] It can be seen that how to balance the brightness and color of the stitched images to improve the user's visual experience is a technical problem that people in this field urgently need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide an image processing method, device, equipment and medium to solve the technical problem that the brightness and color of the spliced ​​images are unbalanced, resulting in a low user visual experience.

[0005] In order to solve the above technical problems, the present invention provides an image processing method, comprising:

[0006] Obtain the average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors; wherein the visual sensor is installed on the vehicle;

[0007] Determine the gain value of each YUV channel of the second visual sensor according to the average value of each channel on the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor;

[0008] Using a first visual sensor to perform texture sampling on a first area and obtain first RGB three-channel data at each position of the first area; and using a second visual sensor to perform texture sampling on a second area, obtain initial RGB three-channel data at each position of the second area, convert the initial RGB three-channel data into YUV three-channel data, and determine second RGB three-channel data according to the YUV three-channel data and corresponding gain values ​​of each YUV channel;

[0009] The first region is colored according to the data of the first RGB three channels, and the second region is colored according to the data of the second RGB three channels, so as to obtain an image around the vehicle.

[0010] Preferably, there are at least four visual sensors, and there is at least one visual sensor in each direction of the vehicle; determining the overlapping area of ​​the fields of view of different visual sensors includes:

[0011] Acquire initial regions of the visual sensor in each direction; wherein each of the initial regions is located in a shooting direction of the visual sensor in each direction, and each of the initial regions includes a first boundary at a first distance from the visual sensor and a second boundary at a second distance from the visual sensor;

[0012] Acquire an initial overlapping area formed by the intersection of the initial areas of the visual sensors in two adjacent directions;

[0013] The overlapping areas of the visual fields of the visual sensors in different directions are determined according to the initial overlapping areas.

[0014] Preferably, determining the overlapping areas of the visual fields of the visual sensors in different directions according to the initial overlapping areas comprises:

[0015] Obtaining the coordinates of each initial overlapping area in the frame coordinate system of the visual sensor in two corresponding adjacent directions;

[0016] Determine the situation of each initial overlapping area on the image corresponding to the visual sensor according to the coordinates of each initial overlapping area on the frame coordinate system of the visual sensor in two corresponding adjacent directions;

[0017] If it is detected that the initial overlapping area is unobstructed on the image corresponding to the visual sensor, the overlapping area is used as the visual field overlapping area of ​​the visual sensor in different directions;

[0018] If it is detected that the overlapping area is blocked on the image corresponding to the visual sensor, the first distance and the second distance are adjusted, the initial overlapping area is re-determined, and the overlapping areas of the visual sensor's field of view in different directions are determined based on the initial overlapping area.

[0019] Preferably, determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor comprises:

[0020] The gain values ​​of each YUV channel of the second vision sensor are determined according to the mean values ​​of each channel on the overlapping area of ​​the current field of view collected by the first vision sensor and the mean values ​​of each channel on the overlapping area of ​​the current field of view collected by the second vision sensor and by using the gradient descent method.

[0021] Preferably, determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor comprises:

[0022] Determine a first gain value of each YUV channel of the second vision sensor according to the average value of each channel on the overlapping area of ​​the first field of view collected by the second vision sensor and the average value of each channel on the overlapping area of ​​the first field of view collected by the first vision sensor located in the first direction;

[0023] Determine the second gain value of each YUV channel of the second vision sensor according to the average value of each channel on the second field of view overlapping area collected by the second vision sensor and the average value of each channel on the second field of view overlapping area collected by the first vision sensor located in the second direction; wherein the first direction and the second direction are both adjacent to the direction where the second vision sensor is located;

[0024] The first gain value and the second gain value of each YUV channel are averaged respectively, and each average is used as the gain value of the corresponding channel of the second visual sensor.

[0025] Preferably, before obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors, the method further includes:

[0026] Obtain the overlapping area of ​​vision of different visual sensors through the central processor;

[0027] After obtaining the overlapping areas of the visual fields of different visual sensors through the central processor, the method further includes:

[0028] Receiving, by a digital signal processor, overlapping areas of vision fields of different visual sensors sent by the central processor;

[0029] The method of obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors includes:

[0030] The digital signal processor is used to obtain the mean values ​​of the YUV channels in the overlapping areas of the visual fields of different visual sensors;

[0031] After obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors by the digital signal processor, the method further includes:

[0032] The central processor receives the average values ​​of each YUV channel on the overlapping area of ​​the fields of view of different visual sensors sent by the digital signal processor;

[0033] The step of determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor comprises:

[0034] In the central processor, the gain value of each YUV channel of the second visual sensor is determined according to the average value of each channel on the overlapping area of ​​the current visual field respectively collected by the first visual sensor and the second visual sensor;

[0035] After the central processor determines the gain value of each YUV channel of the second visual sensor according to the average value of each channel on the current visual field overlap area respectively collected by the first visual sensor and the second visual sensor, the method further includes:

[0036] The gain values ​​of each YUV channel of the second visual sensor are sent to the graphics processing unit through the central processing unit.

[0037] Preferably, the central processor sending the overlapping areas of the fields of view of different visual sensors to the digital signal processor comprises:

[0038] Divide the image corresponding to each visual sensor into a plurality of squares;

[0039] Get the target squares corresponding to the overlapping areas of each field of view;

[0040] Sending the parameters of each target square to the digital signal processor; wherein the parameters of the target square at least include the size of the target square and the position of the target square on the image corresponding to the visual sensor;

[0041] The method of obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors by using a digital signal processor includes:

[0042] Calculate the mean value of each YUV channel of each target square by the digital signal processor;

[0043] Get the mean value of each YUV channel of all target squares corresponding to each overlapping area of ​​the field of view.

[0044] In order to solve the above technical problem, the present invention further provides an image processing device, comprising:

[0045] An acquisition module is used to acquire the average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors; wherein the visual sensor is installed on the vehicle;

[0046] A determination module, used to determine the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor; wherein the first visual sensor and the second visual sensor have an overlapping area of ​​the field of view;

[0047] a sampling and acquisition module, configured to perform texture sampling on a first area using a first visual sensor and acquire data of first RGB three channels at each position of the first area; and perform texture sampling on a second area using a second visual sensor, acquire data of initial RGB three channels at each position of the second area, convert the data of the initial RGB three channels into data of YUV three channels, and determine data of second RGB three channels according to the data of the YUV three channels and the gain values ​​of the corresponding YUV channels;

[0048] The coloring module is used to color the first area according to the data of the first RGB three channels and to color the second area according to the data of the second RGB three channels, so as to obtain an image around the vehicle.

[0049] In order to solve the above technical problems, the present invention further provides an image processing device, comprising:

[0050] Memory for storing computer programs;

[0051] A processor is used to implement the steps of the above-mentioned image processing method when executing the computer program.

[0052] In order to solve the above technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above image processing method are implemented.

[0053] In the image processing method provided by the present invention, first, the gain value of each YUV channel of the second visual sensor is determined according to the average value of each channel on the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor, and then when coloring the first area, the first visual sensor is used to sample the texture of the first area, and the first area is directly colored using the sampled RGB color; when coloring the second area, the RGB value obtained after the second visual sensor samples the texture of the second area is converted into a YUV value, and calibration is performed using the gain value of each YUV channel of the second visual sensor determined previously, so as to determine the final RGB color of the second area, and the second area is colored using the final RGB color. It can be seen that in this method, the gain value is applied to the coloring process to achieve balanced processing of brightness and color, thereby improving the user's visual experience. In addition, when determining the overlapping area of ​​the fields of view of different visual sensors, the overlapping area of ​​the fields of view of the visual sensors in different directions is determined according to the initial overlapping area formed by the intersection of the initial areas of the visual sensors in two adjacent directions (including the first boundary at a first distance from the visual sensor and the second boundary at a second distance from the visual sensor). That is, the overlapping area of ​​the field of view is limited to the area formed by the intersection of the initial areas of the visual sensors in two adjacent directions, and only the pixels in this area are calculated. The amount of calculation is relatively small, and no complex system is required. It can also be implemented on a small computing power platform.

[0054] In addition, the present invention also provides an image processing device, an image processing equipment and a computer-readable storage medium, which have the same or corresponding technical features as the above-mentioned image processing method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of determining a visual field overlap area provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of a finally selected ROI area corresponding to a lens provided in an embodiment of the present invention;

[0059] Figure 4A flowchart of a method for determining a current gain value of a left / right lens provided by an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of areas divided during rendering provided by an embodiment of the present invention;

[0061] Figure 6 A flow chart of a method for determining the color of each region when rendering provided by an embodiment of the present invention;

[0062] Figure 7 A schematic diagram of a square selected on a lens image provided by an embodiment of the present invention;

[0063] Figure 8 A schematic diagram of a method for performing image processing by using different core processors provided by an embodiment of the present invention;

[0064] Fig. 9 A structural diagram of an image processing device provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The core of the present invention is to provide an image processing method, device, equipment and medium to solve the technical problem that the brightness and color of the spliced ​​image are unbalanced, resulting in a low user visual experience.

[0067] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0068] S10: Obtain the average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors;

[0069] S11: determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor;

[0070] S12: using the first visual sensor to perform texture sampling on the first area and obtain first RGB three-channel data of each position of the first area; and using the second visual sensor to perform texture sampling on the second area, obtain initial RGB three-channel data of each position of the second area, convert the initial RGB three-channel data into YUV three-channel data, and determine the second RGB three-channel data according to the YUV three-channel data and the corresponding gain values ​​of each YUV channel;

[0071] S13: Coloring the first area according to the data of the first RGB three channels and coloring the second area according to the data of the second RGB three channels to obtain an image around the vehicle.

[0072] The visual sensor (camera) is installed on the vehicle. There is no limitation on the number and position of the visual sensors installed on the vehicle, but in order to obtain the entire image around the vehicle, there are at least four visual sensors, and there is at least one visual sensor in each direction of the vehicle. In order to achieve brightness and color balance processing, in the embodiment of the present invention, the values ​​of the three channels of YUV (wherein the Y component represents the brightness information of the image, and the U and V components represent the color information of the image) collected by the first visual sensor are used as a reference to determine the gain values ​​of each channel of YUV of the second visual sensor. That is, the first visual sensor can also be called the main visual sensor, and the second visual sensor can be called the visual sensor to be calibrated. When selecting the first visual sensor and the second visual sensor, first, it is necessary to ensure that there is an overlapping field of view of the two visual sensors, and then randomly select one visual sensor from the two visual sensors with the overlapping field of view as the main visual sensor, and the other visual sensor is the visual sensor to be calibrated.

[0073] Specifically, in determining the gain value of the three YUV channels of the second visual sensor, first, obtain the mean value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors, and then determine the gain value of each YUV channel of the second visual sensor according to the mean value of each channel in the overlapping area of ​​the current field of view collected by the first visual sensor and the mean value of each channel in the overlapping area of ​​the current field of view collected by the second visual sensor. By applying the gain value to the color of the texture sampled by the second visual sensor, the final output color is obtained, so that the color of the texture sampled by the first visual sensor is the same as or similar to the final output color. There is no limitation on the specific method of determining the gain value according to the mean value of each channel in the overlapping area of ​​the current field of view collected by the first visual sensor and the mean value of each channel in the overlapping area of ​​the current field of view collected by the second visual sensor, such as using the gradient descent method.

[0074] After determining the gain value of each YUV channel of the second visual sensor, use the fragment shader for coloring. Different areas are divided in the area outside the vehicle body, such as one area each in the front, back, left and right of the vehicle body. The first visual sensor is used to perform texture sampling on the first area, and the second visual sensor is used to perform texture sampling on the second area. Since the first visual sensor is a visual sensor that does not require calibration, the data of the three channels of red (Red), green (Green) and blue (Blue) (RGB for short) at each position obtained by the first visual sensor after performing texture sampling on the first area is the data for coloring at each position of the first area. The second visual sensor is a visual sensor that needs to be calibrated, so the initial RGB three-channel data of each position obtained by the second visual sensor after performing texture sampling on the second area needs to be converted into YUV three-channel data, and the data of the second RGB three-channel is determined according to the data of the YUV three-channel and the corresponding gain values ​​of each YUV channel, so as to use the data of the second RGB three-channel to color the second area.

[0075] The following describes a process for determining the data of the second RGB three channels (ie, the color output value of the second visual sensor after gain calibration) (ie, the RGB conversion formula).

[0076] First, the initial RGB three-channel data (r, g, and b) is converted to Y / U / V (Y / U / V values ​​obtained based on r, g, and b) three-channel data. The process is as follows:

[0077] Y =0.25881127 * (r + 222.625) + 0.50766827 * (g - 135.875) +0.09838532 * (b + 276.109375);

[0078] U = -0.14846538 * (r + 222.625) - 0.29122056 * (g - 135.875) +0.43968594 * (b + 276.109375);

[0079] V = 0.43968594 * (r + 222.625) - 0.36830835 * (g - 135.875) -0.07137759 * (b + 276.109375);

[0080] Then, the data of the second RGB three channels are determined according to the data of the Y / U / V three channels and the gain values ​​of the corresponding YUV channels. The process is as follows:

[0081] R = 1.15625 * Y *u_gainY+ 1.59375 * V*u_gainV - 222.625;

[0082] G = 1.15625 * Y* u_gainY- 0.390625 * U *u_gainU- 0.8125 * V*u_gianV +135.875;

[0083] B = 1.15625 * Y* u_gainY+ 2.015625 * U*u_gainU - 276.109375;

[0084] Among them, u_gainY / u_gainU / u_gainV represent the calculated gain values ​​of the three channels Y / U / V, and R / G / B represent the final color output values ​​after gain calibration.

[0085] In the image processing method provided by the embodiment of the present invention, first, the gain value of each YUV channel of the second visual sensor is determined according to the mean value of each channel on the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor, and then when coloring the first area, the first visual sensor is used to sample the texture of the first area, and the first area is directly colored using the sampled RGB color; and when coloring the second area, the RGB value obtained after the second visual sensor samples the texture of the second area is converted into a YUV value, and the gain value of each YUV channel of the second visual sensor determined before is used for calibration, so as to determine the final RGB color of the second area, and the second area is colored using the final RGB color. It can be seen that in this method, the gain value is applied to the coloring process, which realizes the balanced processing of brightness and color and improves the user's visual experience.

[0086] In practice, the visual overlap area between different visual sensors is relatively large, resulting in a large amount of computation. Therefore, in some embodiments, determining the visual overlap area of ​​different visual sensors includes:

[0087] Acquire initial regions of the visual sensor in each direction; wherein each initial region is located in a shooting direction of the visual sensor in each direction, and each initial region includes a first boundary at a first distance from the visual sensor and a second boundary at a second distance from the visual sensor;

[0088] Obtain an initial overlapping area formed by the intersection of initial areas of the visual sensors in two adjacent directions;

[0089] The overlapping areas of the visual fields of the visual sensors in different directions are determined according to the initial overlapping areas.

[0090] The following takes the example of installing a visual sensor on each of the four sides of the vehicle to illustrate the method of determining the overlapping area of ​​vision (ie, region of interest (ROI)) of the present invention. Figure 2 A schematic diagram of determining a visual field overlap area provided by an embodiment of the present invention. Figure 2 As shown, it includes four lenses, front, back, left and right. The initial area of ​​the front lens is the area between a and b, the initial area of ​​the left lens is the area between c and d, the initial area of ​​the rear lens is the area between e and f, and the initial area of ​​the right lens is the area between g and h. The area between a and b and the area between c and d intersect to form an area ; The area between c and d and the area between e and f intersect to form the area ; The area between e and f and the area between g and h intersect to form the area ; The area between g and h intersects with the area between a and b to form the area .

[0091] It is worth noting that, in practice, the determined initial overlapping area may be blocked, which may affect the calculation result of the gain value. Therefore, after obtaining the initial overlapping area formed by the intersection of the initial areas of the visual sensors in two adjacent directions, the visual field overlapping areas of the visual sensors in different directions are further determined based on the initial overlapping area. Specifically, determining the visual field overlapping areas of the visual sensors in different directions based on the initial overlapping area includes:

[0092] Obtaining the coordinates of each initial overlapping area in the frame coordinate system of the visual sensor in two corresponding adjacent directions;

[0093] Determine the situation of each initial overlapping area on the image corresponding to the visual sensor according to the coordinates of each initial overlapping area on the frame coordinate system of the visual sensor in two corresponding adjacent directions;

[0094] If it is detected that the initial overlapping area is unobstructed on the image corresponding to the visual sensor, the overlapping area is used as the visual field overlapping area of ​​the visual sensor in different directions;

[0095] If it is detected that the overlapping area is blocked on the image corresponding to the visual sensor, the first distance and the second distance are adjusted, the initial overlapping area is re-determined, and the visual field overlapping areas of the visual sensors in different directions are determined according to the initial overlapping area.

[0096] In the vehicle coordinate system in real space, select Figure 2 The four overlapping regions shown in the figure can be adjusted according to the size of the vehicle. The ROI selection is performed as follows:

[0097] 1. Set an initial area for each lens, starting from the first distance from the lens and ending at the second distance along the lens shooting direction;

[0098] 2. Intersect the strip area of ​​each lens obtained in step 1 with the strip area of ​​its adjacent lens to obtain four rectangular areas, which are the initial overlapping areas of the lenses, i.e., the initial ROI areas;

[0099] 3. Convert the initial ROI area into coordinates on the frame coordinate system of each lens through calibration parameters, and the area Convert to the front and right lens frame coordinate systems respectively, area Convert to the front and left lens frame coordinate systems respectively, and so on.

[0100] 4. Bring this coordinate back to the original lens fisheye image in turn to check the actual effect of the initial ROI area in each scene, and ensure that the initial ROI area is not blocked by the lens frame and is not in the shadow area of ​​the vehicle body.

[0101] 5. Repeat the above process to determine the appropriate first distance and second distance until a reasonable ROI area is determined.

[0102] The above steps 1 and 2 can be considered as roughly selecting the ROI area around the vehicle body, and steps 3 to 5 can be considered as calculating the final ROI area based on the roughly selected ROI area around the vehicle body and combining the calibration parameters of the surround view system. Figure 3 A schematic diagram of a finally selected ROI area corresponding to a lens provided in an embodiment of the present invention. Figure 3 As shown, two ROI areas corresponding to the lens are selected.

[0103] Compared with pixel-by-pixel processing that results in increased computational complexity, or involving neural networks and resulting in complex design systems, the method provided in this embodiment only calculates the pixels within the ROI area, so the amount of computation is relatively small and does not require a complex system, and can be implemented on a low-computing power platform.

[0104] When determining the gain values ​​of each YUV channel of the second vision sensor according to the average value of each channel on the overlapping area of ​​the current field of view collected by the first vision sensor and the average value of each channel on the overlapping area of ​​the current field of view collected by the second vision sensor, the gradient descent method is used in the embodiment of the present invention.

[0105] The method of determining the gain values ​​of the three channels in a specific ROI area based on the gradient descent method specifically includes the following process:

[0106] 1. First, define the deviation function:

[0107] ;

[0108] Wherein, E represents the deviation function, m represents the average value of any one of the three channels Y / U / V of the first visual sensor, n represents the average value of the corresponding channel of the second visual sensor, and x represents the gain value of the corresponding channel of the second visual sensor.

[0109] 2. Based on gradient descent, obtain the new gain value of the channel corresponding to the lens to be calibrated. The opposite direction of the gradient is the fastest direction in which the function value decreases, so the new gain value is taken as follows:

[0110] ;

[0111] Where k is the step size coefficient, Indicates the gain value of the previous frame.

[0112] When determining the gain value of each YUV channel of the second visual sensor, in order to improve the accuracy of the determined gain value, determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel on the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor includes:

[0113] Determine a first gain value of each YUV channel of the second vision sensor according to the average value of each channel on the overlapping area of ​​the first field of view collected by the second vision sensor and the average value of each channel on the overlapping area of ​​the first field of view collected by the first vision sensor located in the first direction;

[0114] Determine the second gain value of each YUV channel of the second vision sensor according to the average value of each channel on the second field of view overlapping area collected by the second vision sensor and the average value of each channel on the second field of view overlapping area collected by the first vision sensor located in the second direction; wherein the first direction and the second direction are both adjacent to the direction where the second vision sensor is located;

[0115] The first gain value and the second gain value of each YUV channel are averaged respectively, and each average is used as the gain value of the corresponding channel of the second visual sensor.

[0116] Still Figure 2 For example, the front lens and the rear lens are used as the first visual sensor, that is, the front lens and the rear lens do not need to be calibrated, and the left lens and the right lens are used as the second visual sensor, that is, the left lens and the right lens need to be calibrated. When calculating the gain value of the left lens, the corresponding first visual sensors are the front lens and the rear lens respectively; when the first visual sensor is the front lens, the first visual field overlap area is the area ; When the first visual sensor is the rear lens, the second visual field overlap area is the area When calculating the gain value of the right lens, the corresponding first visual sensors are also the front lens and the rear lens respectively; when the first visual sensor is the front lens, the first visual field overlap area is the area ; When the first visual sensor is the rear lens, the second visual field overlap area is the area . Figure 4 A flowchart of a method for determining the current gain value of a left / right lens provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the method includes:

[0117] S14: receiving the average of each channel of the four lenses;

[0118] S15: taking the front lens as the main lens, and obtaining the current gain of the left lens based on the gain value calculation formula;

[0119] S16: taking the front lens as the main lens, and calculating the current gain of the right lens based on the gain value calculation formula;

[0120] S17: taking the subsequent lens as the main lens, obtaining the current gain of the left lens based on the gain value calculation formula;

[0121] S18: taking the rear lens as the main lens, and obtaining the current gain of the right lens based on the gain value calculation formula;

[0122] S19: taking the average of the gain values ​​of the two left / right lenses as the final current gain of the left / right lens.

[0123] Then the process may return to step S14.

[0124] After obtaining the gain value, the gain value is applied to the fragment shader code of the final rendering of the surround stitching to achieve balanced processing of brightness and color. Figure 2 For example, when rendering, different areas are divided, such as Figure 5 As shown, Figure 5 A schematic diagram of a region divided during rendering provided by an embodiment of the present invention. The front region, rear region, left region, and right region are divided. Four lens images are used Figure 5 Texture sampling is performed in four areas. Figure 6 A flow chart of a method for determining the color of each region when rendering is provided in an embodiment of the present invention, such as Figure 6 As shown, the method includes:

[0125] S20: using the front lens data to perform texture sampling on the front area to obtain a sampled color of the front area;

[0126] S21: Using the left lens data to perform texture sampling on the left area, and converting the sampled RGB value using the RGB conversion formula to obtain a value as the final sampled color of the left area;

[0127] S22: Using the rear lens data to perform texture sampling on the rear area to obtain a sampled color of the rear area;

[0128] S23: Using the right lens data to perform texture sampling on the right region, and converting the sampled RGB value using an RGB conversion formula to obtain a value as the final sampled color of the right region.

[0129] In the method provided in the embodiment of the present invention, the final ROI range is determined based on the calibration parameters of the surround view system and the selected vehicle body surrounding area, and the mean of the Y channel and UV channel is calculated only for the pixels within the ROI range. The front or rear lens is used as the reference lens, and the gradient descent method is used to iteratively calculate the gain values ​​required for the remaining lens images. The gain value is applied to the fragment shader code of the final rendering of the surround view splicing to achieve balanced processing of brightness and color.

[0130] In the implementation, in order to balance the overall system load, different operations are assigned to different core processors. Specifically, before obtaining the average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors, it also includes:

[0131] The overlapping area of ​​the visual field of different visual sensors is obtained through the central processing unit (CPU);

[0132] After obtaining the overlapping areas of vision of different visual sensors through the central processor, it also includes:

[0133] Receive the overlapping areas of vision of different visual sensors sent by the central processor through the digital signal processor (DSP);

[0134] Obtaining the mean of each YUV channel in the overlapping area of ​​the field of view of different visual sensors includes:

[0135] The average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors is obtained through a digital signal processor;

[0136] After obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the field of view of different visual sensors through the digital signal processor, it also includes:

[0137] The central processor receives the average value of each YUV channel on the overlapping area of ​​the field of view of different visual sensors sent by the digital signal processor;

[0138] Determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel on the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor includes:

[0139] In the central processing unit, the gain value of each YUV channel of the second visual sensor is determined according to the average value of each channel on the overlapping area of ​​the current visual field respectively collected by the first visual sensor and the second visual sensor;

[0140] After the gain value of each YUV channel of the second visual sensor is determined in the central processor according to the average value of each channel on the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor, the method further includes:

[0141] The gain values ​​of each YUV channel of the second visual sensor are sent to a graphics processing unit (GPU) through the central processing unit.

[0142] After that, rendering takes place in the graphics processing unit.

[0143] In order to reduce the number of parameters transmitted to the DSP, the CPU sends the overlapping areas of the fields of view of different vision sensors to the DSP including:

[0144] Divide the image corresponding to each visual sensor into a plurality of squares;

[0145] Get the target squares corresponding to the overlapping areas of each field of view;

[0146] Sending the parameters of each target square to a digital signal processor; wherein the parameters of the target square at least include the size of the target square and the position of the target square on the image corresponding to the visual sensor;

[0147] The mean values ​​of each YUV channel in the overlapping area of ​​the field of view of different visual sensors are obtained by the digital signal processor, including:

[0148] The mean value of each YUV channel of each target square is calculated by a digital signal processor;

[0149] Get the mean value of each YUV channel of all target squares corresponding to each overlapping area of ​​the field of view.

[0150] Specifically, if the lens image is divided into squares of 16×16 pixels, the squares within the coordinate interval in the frame coordinate system calculated during the ROI selection process are selected and passed as parameters to the digital signal processor. Figure 7 A schematic diagram of selecting a square on a lens image provided by an embodiment of the present invention. Figure 7 As shown in the irregular graphics.

[0151] The digital signal processor calculates the average of the three channels Y / U / V for each incoming square, and then obtains the average of the three channels of all incoming squares.

[0152] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 8 A schematic diagram of a method for performing image processing using different core processors provided by an embodiment of the present invention. Figure 8 As shown, the process includes:

[0153] 1. In the central processing unit, the ROI area is selected by calibration parameters, and the selected ROI area is passed to the digital signal processor as a configuration parameter;

[0154] 2. Calculate the mean of the three channels Y / U / V in the digital signal processor and transmit the mean of each channel to the central processor;

[0155] 3. Calculate the Y / U / V gain of a specific ROI area in the CPU and transmit the gain value to the graphics processing unit;

[0156] 4. In the graphics processing unit, the gain value is applied to the fragment shader for rendering.

[0157] Specifically, when image processing is performed through different core processors, the process is as follows:

[0158] 1. Roughly select the ROI area around the vehicle body in the central processor, combine the calibration parameters of the surround view system, calculate the final ROI, and convert it to the fisheye images of the four lenses;

[0159] Second, the ROI area on the fisheye image is configured as a parameter to the digital signal processor, and the four fisheye images are input into the digital signal processor to calculate the mean value of the Y / U / V channels in the ROI area;

[0160] 3. The calculated mean values ​​are passed back to the CPU. For the intersection area in the actual space, two adjacent lenses will produce different mean values ​​of Y / U / V on their respective ROIs. Select a lens as the main lens, and use the gradient descent method to iteratively obtain the gain values ​​of the remaining lenses to be calibrated;

[0161] Fourth, the gain value is passed as a variable to the graphics processing unit, acts on the fragment shader, and the color directly obtained by texture sampling is processed through the gain value to obtain the final balanced color.

[0162] In the method provided by the embodiment of the present invention, only the pixels within the ROI are calculated, the amount of calculation is relatively small, and no complex system is required. It can also be implemented on a small computing power platform; the brightness and color can be balanced simultaneously or separately; and different operations are assigned to different core processors to balance the overall system load, and finally the brightness and color balance of the surround stitching image is achieved, so that the visual effect of the stitched image is more natural, and the user's visual experience is improved.

[0163] In the above embodiments, the image processing method is described in detail, and the present invention also provides corresponding embodiments of the image processing apparatus and the image processing device. It should be noted that the present invention describes the embodiments of the apparatus part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.

[0164] An image processing device provided by an embodiment of the present invention, based on the perspective of functional modules, includes:

[0165] An acquisition module is used to acquire the average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors; wherein the visual sensor is installed on the vehicle;

[0166] A determination module, used to determine the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor; wherein the first visual sensor and the second visual sensor have an overlapping area of ​​the field of view;

[0167] The sampling and acquisition module is used to perform texture sampling on the first area using the first visual sensor and acquire the first RGB three-channel data of each position of the first area; and perform texture sampling on the second area using the second visual sensor to acquire the initial RGB three-channel data of each position of the second area, convert the initial RGB three-channel data into YUV three-channel data, and determine the second RGB three-channel data according to the YUV three-channel data and the corresponding gain values ​​of each YUV channel;

[0168] The coloring module is used to color the first area according to the data of the first RGB three channels and to color the second area according to the data of the second RGB three channels to obtain an image around the vehicle.

[0169] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0170] Fig. 9 This is a structural diagram of an image processing device provided by another embodiment of the present invention. This embodiment is based on the hardware perspective, such as Fig. 9 As shown, the image processing device includes:

[0171] A memory 20, used for storing computer programs;

[0172] The processor 21 is used to implement the steps of the image processing method mentioned in the above embodiment when executing the computer program.

[0173] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP, Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU, which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0174] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the image processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the above-mentioned image processing method, etc.

[0175] In some embodiments, the image processing device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0176] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the image processing device, and may include more or fewer components than those shown in the figure.

[0177] The image processing device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: an image processing method, and the effect is the same as above.

[0178] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0179] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0180] The computer-readable storage medium provided by the present invention includes the above-mentioned image processing method, and the effect is the same as above.

[0181] The above is a detailed introduction to an image processing method, device, equipment and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0182] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. An image processing method, characterized in that: include: Obtain the average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors; wherein the visual sensor is installed on the vehicle; Determine the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view collected by the first visual sensor and the second visual sensor respectively; wherein the first visual sensor and the second visual sensor have an overlapping area of ​​the field of view; Using a first visual sensor to perform texture sampling on a first area and obtain first RGB three-channel data of each position of the first area; and using a second visual sensor to perform texture sampling on a second area, obtain initial RGB three-channel data of each position of the second area, convert the initial RGB three-channel data into YUV three-channel data, and determine the second RGB three-channel data according to the YUV three-channel data and the corresponding gain values ​​of each YUV channel; wherein the first area is a field of view area in the shooting direction of the first visual sensor, and the second area is a field of view area in the shooting direction of the second visual sensor; The first region is colored according to the data of the first RGB three channels, and the second region is colored according to the data of the second RGB three channels, so as to obtain an image around the vehicle.

2. The image processing method according to claim 1, characterized in that: There are at least four visual sensors, and there is at least one visual sensor in each direction of the vehicle; Determining the overlapping areas of the vision sensors includes: Acquire initial regions of the visual sensor in each direction; wherein each of the initial regions is located in a shooting direction of the visual sensor in each direction, and each of the initial regions includes a first boundary at a first distance from the visual sensor and a second boundary at a second distance from the visual sensor; Acquire an initial overlapping area formed by the intersection of the initial areas of the visual sensors in two adjacent directions; The overlapping areas of the visual fields of the visual sensors in different directions are determined according to the initial overlapping areas.

3. The image processing method according to claim 2, characterized in that: Determining the overlapping areas of the visual field of the visual sensors in different directions according to the initial overlapping areas comprises: Obtaining the coordinates of each initial overlapping area in the frame coordinate system of the visual sensor in two corresponding adjacent directions; Determine the situation of each initial overlapping area on the image corresponding to the visual sensor according to the coordinates of each initial overlapping area on the frame coordinate system of the visual sensor in two corresponding adjacent directions; If it is detected that the initial overlapping area is unobstructed on the image corresponding to the visual sensor, the overlapping area is used as the visual field overlapping area of ​​the visual sensor in different directions; If it is detected that the overlapping area is blocked on the image corresponding to the visual sensor, the first distance and the second distance are adjusted, the initial overlapping area is re-determined, and the overlapping areas of the visual sensor's field of view in different directions are determined based on the initial overlapping area.

4. The image processing method according to claim 1, characterized in that: The step of determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor comprises: The gain values ​​of each YUV channel of the second visual sensor are determined according to the mean values ​​of each channel on the overlapping area of ​​the current visual field respectively collected by the first visual sensor and the second visual sensor and by using the gradient descent method.

5. The image processing method according to claim 4, characterized in that: The step of determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor comprises: Determine a first gain value of each YUV channel of the second vision sensor according to the average value of each channel on the overlapping area of ​​the first field of view collected by the second vision sensor and the average value of each channel on the overlapping area of ​​the first field of view collected by the first vision sensor located in the first direction; Determine the second gain value of each YUV channel of the second vision sensor according to the average value of each channel on the second field of view overlapping area collected by the second vision sensor and the average value of each channel on the second field of view overlapping area collected by the first vision sensor located in the second direction; wherein the first direction and the second direction are both adjacent to the direction where the second vision sensor is located; The first gain value and the second gain value of each YUV channel are averaged respectively, and each average is used as the gain value of the corresponding channel of the second visual sensor.

6. The image processing method according to any one of claims 1 to 5, characterized in that: Before obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors, the method further includes: Obtain the overlapping area of ​​vision of different visual sensors through the central processor; After obtaining the overlapping areas of the visual fields of different visual sensors through the central processor, the method further includes: Receiving, by a digital signal processor, overlapping areas of vision fields of different visual sensors sent by the central processor; The method of obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors includes: The digital signal processor is used to obtain the mean values ​​of the YUV channels in the overlapping areas of the visual fields of different visual sensors; After obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors by the digital signal processor, the method further includes: The central processor receives the average values ​​of each YUV channel on the overlapping area of ​​the fields of view of different visual sensors sent by the digital signal processor; The step of determining the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor comprises: In the central processor, the gain value of each YUV channel of the second visual sensor is determined according to the average value of each channel on the overlapping area of ​​the current visual field respectively collected by the first visual sensor and the second visual sensor; After the central processor determines the gain value of each YUV channel of the second visual sensor according to the average value of each channel on the current visual field overlap area respectively collected by the first visual sensor and the second visual sensor, the method further includes: The gain values ​​of each YUV channel of the second visual sensor are sent to the graphics processing unit through the central processing unit.

7. The image processing method according to claim 6, characterized in that: The central processor sends the overlapping areas of the fields of view of different visual sensors to the digital signal processor, including: Divide the image corresponding to each visual sensor into a plurality of squares; Get the target squares corresponding to the overlapping areas of each field of view; Sending the parameters of each target square to the digital signal processor; wherein the parameters of the target square at least include the size of the target square and the position of the target square on the image corresponding to the visual sensor; The method of obtaining the mean values ​​of each YUV channel in the overlapping area of ​​the fields of view of different visual sensors by using a digital signal processor includes: Calculate the mean value of each YUV channel of each target square by the digital signal processor; Get the mean value of each YUV channel of all target squares corresponding to each overlapping area of ​​the field of view.

8. An image processing device, characterized in that: include: An acquisition module is used to acquire the average value of each YUV channel in the overlapping area of ​​the field of view of different visual sensors; wherein the visual sensor is installed on the vehicle; A determination module, used to determine the gain value of each YUV channel of the second visual sensor according to the average value of each channel in the overlapping area of ​​the current field of view respectively collected by the first visual sensor and the second visual sensor; wherein the first visual sensor and the second visual sensor have an overlapping area of ​​the field of view; A sampling and acquisition module, used to perform texture sampling on a first area using a first visual sensor and acquire first RGB three-channel data of each position of the first area; and perform texture sampling on a second area using a second visual sensor, acquire initial RGB three-channel data of each position of the second area, convert the initial RGB three-channel data into YUV three-channel data, and determine the second RGB three-channel data according to the YUV three-channel data and the corresponding gain values ​​of each YUV channel; wherein the first area is a field of view area in the shooting direction of the first visual sensor, and the second area is a field of view area in the shooting direction of the second visual sensor; The coloring module is used to color the first area according to the data of the first RGB three channels and to color the second area according to the data of the second RGB three channels, so as to obtain an image around the vehicle.

9. An image processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images