Image processing method and device, equipment and computer storage medium

Abstract

The invention discloses an image processing method and device, equipment and a computer storage medium. The method comprises the steps that the color richness corresponding to a first image and the number of white point pixels corresponding to the first image are determined; wherein the first image is any image in the video images; under the condition that the color richness is smaller than or equal to a color richness threshold value and the number of the white point pixels is smaller than or equal to a number threshold value, obtaining pose data corresponding to the first image; determining a reference image corresponding to the first image in the second image according to the pose data corresponding to the first image; wherein the second image is the other image except the first image in the video image; and performing image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and apparatus, equipment and computer storage medium. Background Art

[0002] In image processing, finding white points (i.e., pure white areas in an image) and performing white balance based on the gray world assumption are two commonly used techniques for adjusting the color balance of an image to make it closer to the real colors perceived by the human eye.

[0003] Among them, the method of finding white points is usually to find the point or area closest to white and adjust the overall color of the image accordingly. The grayscale world assumption is to adjust the value of each color channel to the proportion of the average value to achieve white balance. However, for images with monotonous colors or images without white points, the image processing parameters determined by the above common methods are often inaccurate, resulting in poor image processing performance. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and apparatus, a device and a computer storage medium, which can ensure the accuracy of image processing parameters, thereby obtaining a more ideal image processing effect and improving image processing performance.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image;

[0008] When the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the number threshold, acquiring the posture data corresponding to the first image;

[0009] Determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image;

[0010] The first image is processed based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:

[0012] A determination unit, configured to determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image;

[0013] An acquisition unit, configured to acquire the posture data corresponding to the first image when the color richness is less than or equal to a color richness threshold and the number of white point pixels is less than or equal to a number threshold;

[0014] The determination unit is further configured to determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image;

[0015] The acquisition unit is further configured to perform image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0016] In a third aspect, an embodiment of the present application provides a computer device, which includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method of the first aspect is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method of the first aspect described above is implemented.

[0018] The embodiment of the present application provides an image processing method and apparatus, a device and a computer storage medium, which determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image; when the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the number threshold, obtain the posture data corresponding to the first image; determine the reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is another image in the video image other than the first image; perform image processing on the first image based on the image processing parameters of the reference image to obtain the processed image corresponding to the first image. That is to say, in the embodiment of the present application, it can be determined whether the image can calculate accurate image processing parameters according to the color richness and the number of white point pixels of the image. If not, it is necessary to refer to the image processing parameters of other images. For example, the posture data corresponding to the image is used to search for an image with a similar posture when shooting as a reference image, and then perform image processing based on the image processing parameters of the reference image. In this way, the accuracy of the image processing parameters can be ensured, and a more ideal image processing effect can be obtained, thereby improving the image processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 1 ;

[0020] Figure 2 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 2 ;

[0021] Figure 3 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 3 ;

[0022] Figure 4 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 4 ;

[0023] Figure 5 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 5 ;

[0024] Figure 6 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 6 ;

[0025] Figure 7 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 7 ;

[0026] Figure 8 A schematic diagram of an implementation of the image processing method proposed in the embodiment of the present application;

[0027] Fig. 9 A schematic diagram of a scatter plot proposed in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of an implementation of the image processing method proposed in the embodiment of the present application;

[0029] Fig.11 A schematic diagram of the structure of the image processing device proposed in the embodiment of the present application;

[0030] Fig.12 A schematic diagram of the composition structure of the computer device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It is understood that the specific embodiments described herein are only used to explain the difference application, rather than to limit the application. It should also be noted that, for the convenience of description, only the parts that are different from the related applications are shown in the drawings.

[0032] In image processing, finding white points (i.e., pure white areas in an image) and performing white balance based on the gray world assumption are two commonly used techniques for adjusting the color balance of an image to make it closer to the real colors perceived by the human eye.

[0033] The method of finding the white point is mainly based on the analysis of the color distribution in the image, assuming that a certain area in the image is perfectly white (RGB value is close to [255,255,255]). However, since there is rarely true pure white in the image, this method usually finds the point or area closest to white and adjusts the overall color of the image accordingly.

[0034] The grayscale world hypothesis states that in an image under average illumination, the average value of all colors tends to gray (i.e., R=G=B). Based on this hypothesis, white balance can be achieved by calculating the average RGB value of all pixels in the image and then adjusting the value of each color channel to the proportion of the average value.

[0035] Among them, the method of finding the white point is usually to find the point or area closest to white and adjust the overall color of the image accordingly. The grayscale world assumption is to adjust the value of each color channel to the proportion of the average value to achieve white balance. However, for images with monotonous colors or images without white points, the above common methods are often inaccurate and have poor image processing performance.

[0036] In order to solve the above problems, in an embodiment of the present application, it is possible to determine whether the image can calculate accurate image processing parameters based on the color richness and the number of white point pixels of the image. If not, it is necessary to refer to the image processing parameters of other images. For example, the posture data corresponding to the image is used to search for images with similar postures when shooting as reference images, and then image processing is performed based on the image processing parameters of the reference image. In this way, the accuracy of the image processing parameters can be ensured, and a more ideal image processing effect can be obtained, thereby improving the image processing performance.

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] An embodiment of the present application provides an image processing method, which can be applied to an image processing device or an electronic device, and can also be applied to any terminal including the image processing device or the electronic device.

[0039] Below, taking an image processing device as an example, the image processing method proposed in the embodiment of the present application is exemplarily described.

[0040] Furthermore, in the embodiments of the present application, Figure 1Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the image processing method may include the following steps:

[0041] Step 101: Determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image.

[0042] In an embodiment of the present application, the color richness corresponding to the first image and the number of white point pixels corresponding to the first image may be determined first; wherein the first image is any image in the video image.

[0043] It can be understood that, in the embodiments of the present application, the video image may be a sequence of images in a video, wherein the video image may include images corresponding to a time sequence in a video.

[0044] Furthermore, in an embodiment of the present application, the video image may be acquired by collecting in a time series, including a time series of multiple moments, and multiple frames of images may be acquired correspondingly to form a corresponding video.

[0045] It is understandable that in the embodiment of the present application, for any frame of the video image, such as the first image, the color richness corresponding to the first image can be determined, wherein the color richness can determine the color expression of the first image.

[0046] Color richness can be understood as an indicator that describes the color diversity and saturation of an image. It can be quantified through multiple dimensions, including but not limited to one or more of saturation, number of hues, brightness range, color entropy, and color richness index. Among them, the higher the saturation value, the more saturated the color; the number of hues directly reflects the diversity of colors in the image; the larger the brightness range, the higher the layering and richness of the colors; the higher the color entropy, the more uniform the color distribution, thereby increasing the richness and complexity of the colors; the color richness index reflects the vividness of the image colors.

[0047] In practical applications, quantitative indicators of color richness such as saturation, number of hues, brightness range, color entropy, and color richness index can be used alone or in combination to evaluate the color performance of an image, and this application does not make specific limitations.

[0048] Furthermore, in an embodiment of the present application, a method for determining the color richness corresponding to a first image may include: for each pixel in the first image, the scatter coordinates corresponding to the pixel may be first determined based on the image data of the pixel; then, based on the scatter coordinates of all pixels in the first image, the mean coordinates corresponding to the first image may be determined; then, based on the mean coordinates and the scatter coordinates of each pixel, the coordinate distance corresponding to each pixel may be determined; finally, based on the coordinate distance corresponding to each pixel, the color richness may be determined.

[0049] It is understood that in the embodiment of the present application, the image data of a pixel may include but is not limited to the pixel value corresponding to the pixel. For example, the image data of any pixel in the first image may be the pixel value corresponding to the three channels of R, G, and B of the pixel.

[0050] Exemplarily, in some embodiments, when determining the color richness corresponding to the first image, the scatter coordinates corresponding to the pixel, such as (R / G, B / G), can be calculated based on the image data of the pixel, such as the pixel values ​​of the three channels of R, G, and B. That is, the pixel values ​​of the three channels of R, G, and B corresponding to each pixel in the first image can be used to calculate R / G and B / G, and the scatter coordinates (R / G, B / G) corresponding to the pixel can be constructed according to the calculation results. All pixels in the first image are traversed to obtain the scatter coordinates (R / G, B / G) corresponding to all pixels. Then, the mean of the scatter coordinates (R / G, B / G) corresponding to all pixels is calculated to obtain the mean coordinates (ave-x, ave-y) of all scatter coordinates, that is, the mean coordinates corresponding to the first image are calculated. Next, for each pixel in the first image, the Euler distance between the scatter coordinates of the pixel and the mean coordinates is calculated, that is, the coordinate distance corresponding to each pixel is calculated. Finally, mathematical operations are performed based on the coordinate distances corresponding to all pixels in the first image to obtain color richness. For example, the standard deviation of the coordinate distances corresponding to all pixels can be calculated, and the calculated standard deviation d is defined as the color richness of the first image.

[0051] That is to say, in an embodiment of the present application, a scatter plot can be constructed based on the image data of each pixel in the first image, and the scatter coordinates of each pixel in the scatter plot can be determined. Among them, a scatter plot is a graph that shows the distribution of data points in a two-dimensional coordinate system, and each data point is usually represented by a pair of coordinate values ​​(x and y). By plotting these points in the coordinate system, the distribution relationship and trend between the data points can be intuitively seen.

[0052] For example, in an embodiment of the present application, a scatter plot generated based on image data of all pixels in the first image has a horizontal coordinate of R / G and a vertical coordinate of B / G. The more dispersed the points in the scatter plot are, the richer the color of the image can be represented.

[0053] It is understandable that in the embodiment of the present application, for any frame of the video image, such as the first image, the number of white point pixels corresponding to the first image can also be determined. The number of white point pixels can be used to determine the number of white point pixels in the first image.

[0054] White point pixels generally refer to pixels in an image that are extremely bright, close to or equal to white. In a binary image, white point pixels may correspond to pixels whose value is 1 (or close to the maximum value).

[0055] Furthermore, in an embodiment of the present application, when determining the number of white spot pixels corresponding to the first image, for each pixel in the first image, the color parameter corresponding to the pixel can be first determined based on the image data of the pixel; then, based on a preset color threshold and the color parameters of all pixels in the first image, the first image is pixel filtered to obtain filtered pixels; finally, the pixel mean of the filtered pixels can be determined based on the image data of the filtered pixels, and based on the pixel mean of the filtered pixels and the preset pixel threshold, the white spot pixels and the number of white spot pixels in the first image can be determined.

[0056] It is understood that in the embodiments of the present application, the color parameters corresponding to the pixels can be used to determine the color performance of the pixels, wherein the color parameters can be determined by the image data corresponding to the pixels. For example, the color parameters corresponding to the pixels can include R / G and B / G, and the pixel values ​​corresponding to the three channels of R, G, and B can be used to calculate R / G and B / G respectively.

[0057] It is understandable that in the embodiments of the present application, the preset color threshold can be used to determine whether a pixel is overexposed. For example, the preset color threshold and the color parameter corresponding to the pixel can be compared to determine whether the pixel is an overexposed pixel. The preset color threshold may include an upper threshold th1 and a lower threshold th2, and the upper threshold th1 is greater than the lower threshold th2. In other words, a color range is formed by the preset color threshold, and whether the pixel is an overexposed pixel is determined based on whether the color parameter of the pixel falls into the color range.

[0058] It can be understood that, in the embodiments of the present application, pixel screening can be understood as the removal of overexposed pixels, that is, the overexposed pixels in the first image are removed, and the remaining pixels are the screened pixels.

[0059] It can be understood that, in the embodiments of the present application, the preset pixel threshold can be used to determine the white point pixel.

[0060] For example, in some embodiments, when determining the number of white spot pixels corresponding to the first image, the R / G and B / G corresponding to each pixel can be calculated based on the image data of each pixel in the first image, such as the pixel values ​​of the three channels of R, G, and B, that is, the color parameters corresponding to each pixel are obtained; then, the preset color threshold (upper threshold th1 and lower threshold th2) are compared with the color parameters of all pixels in the first image. If the comparison result of the color parameter of a pixel and the preset color threshold does not satisfy th2<R / G<th1 and th2<B / G<th1, then the pixel can be eliminated. Except for the pixel, otherwise, if th2<R / G<th1 and th2<B / G<th1 are satisfied, then the pixel is retained to obtain the filtered pixel; finally, the pixel values ​​of the three channels of R, G, and B of each filtered pixel are averaged to determine the pixel mean ave-rgb of the filtered pixel, and the pixel mean ave-rgb of the filtered pixel is compared with the preset pixel threshold th3. If ave-rgb>th3, then the filtered pixel can be determined as the white point pixel in the first image, and each filtered pixel is traversed to finally determine the number of white point pixels in the first image.

[0061] Step 102: When the color richness is less than or equal to the color richness threshold and the number of white point pixels is less than or equal to the number threshold, obtain the posture data corresponding to the first image.

[0062] In an embodiment of the present application, after determining the color richness corresponding to the first image and the number of white point pixels corresponding to the first image, the color richness and the color richness threshold can be compared, and the number of white point pixels and the number threshold can be compared. If the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the number threshold, then the posture data corresponding to the first image can be further obtained.

[0063] It can be understood that, in the embodiment of the present application, the color richness threshold and the quantity threshold can be used to determine whether to calculate the image processing parameters corresponding to the first image.

[0064] Further, in an embodiment of the present application, when determining whether to calculate the image processing parameters corresponding to the first image through the color richness threshold and the quantity threshold, the color richness threshold and the color richness, the quantity threshold and the number of white point pixels can be compared respectively. If the comparison result is that the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the quantity threshold, then it can be considered that the first image does not meet the conditions for calculating the corresponding image processing parameters, that is, it is considered that the calculated image processing parameters have poor accuracy, and therefore it is determined not to calculate the image processing parameters corresponding to the first image, but to choose to refer to the image processing parameters of other images.

[0065] It is understandable that in the embodiment of the present application, after comparing the color richness and the color richness threshold, if the comparison result is that the color richness is less than or equal to the color richness threshold, then it can be considered that the color richness of the first image is insufficient, and the accuracy of the image processing parameters calculated is poor. After comparing the number of white point pixels and the number threshold, if the comparison result is that the number of white point pixels is less than or equal to the number threshold, then it can be considered that there are no white point pixels in the first image, and the image processing parameters cannot be determined using the white point pixels, that is, the accuracy of the image processing parameters calculated is poor.

[0066] It is understandable that in the embodiment of the present application, if it is determined that the first image does not meet the conditions for calculating the corresponding image processing parameters, then the posture data corresponding to the first image can be obtained first, so that the final reference image can be determined based on the posture data corresponding to the first image. Among them, the posture relationship corresponding to the first image can be used to determine the position and posture of the shooting module when capturing the first image.

[0067] Furthermore, in an embodiment of the present application, a mapping relationship between an image and a posture may be pre-constructed, so that posture data corresponding to the first image may be acquired based on the mapping relationship between the image and the posture.

[0068] Furthermore, in an embodiment of the present application, when a video image corresponding to a time series is captured by a shooting module, the posture data of the shooting module is acquired according to the time series; then, a mapping relationship between the image and the posture can be constructed based on the video image and the posture data of the shooting module.

[0069] It is understandable that in the embodiments of the present application, in the process of video image acquisition and recording by the shooting module, the position and posture of the shooting module can also be determined in real time by the sensor, that is, at the same time, there is a frame of image captured by shooting, and there is also the position and posture of the shooting module detected and calculated, that is, posture data. According to the image and posture data corresponding to each moment in the time series, the mapping relationship between the obtained image and posture can be constructed.

[0070] That is to say, in the embodiment of the present application, the mapping relationship between the image and the posture includes the image and the posture data of the shooting module corresponding to any moment in the time series.

[0071] It is understandable that in the embodiments of the present application, the mapping relationship between the image and the posture can be obtained by the first device from other devices, or can be constructed by the first device. For example, the first device is configured with a shooting module and a sensor. During the process of recording video images through the shooting module, the posture data of the shooting module is synchronously acquired through the sensor, and then the mapping relationship between the image and the posture is constructed based on the correspondence between the video image and the posture data and the time series.

[0072] Furthermore, in the embodiments of the application, the acquisition of the position and posture data of the shooting module can be completed by running a Simultaneous Localization and Mapping (SLAM) algorithm. Among them, the SLAM algorithm is a technology that estimates its own position and posture in real time in an unknown environment through its own sensors (such as laser radar, camera, inertial measurement unit, etc.) and constructs an incremental map of the surrounding environment.

[0073] Based on the SLAM algorithm, the robot locates its own position and posture in an unknown environment by repeatedly observing environmental features and constructs a map of the surrounding environment. The operation of the SLAM algorithm may include the following parts:

[0074] Sensor data processing: Obtain data about the robot’s surrounding environment through sensors such as lidar and cameras.

[0075] Data preprocessing: Perform preprocessing operations such as filtering and denoising on the collected data to improve the quality and accuracy of the data.

[0076] Feature extraction: Extract feature points or feature areas from the preprocessed data. These feature points will be used for subsequent pose estimation and map construction.

[0077] Pose estimation: Use the extracted features and combine them with the motion model to estimate the robot's pose (i.e., position and attitude).

[0078] Map construction: Build an incremental map of the surrounding environment based on the robot’s position and extracted features.

[0079] Backend optimization: The robot’s position and map are further optimized through filtering algorithms or optimization algorithms to improve the accuracy of positioning and map construction.

[0080] Closed-loop detection: Identify locations that the robot has visited before, and optimize and correct the map through closed-loop detection to eliminate accumulated errors.

[0081] Furthermore, in an embodiment of the present application, the mapping relationship between the image and the posture can be expressed as a set of image and posture correspondence tables, wherein at the same time, the image acquired by the shooting module and the posture data of the shooting module have a one-to-one correspondence.

[0082] Exemplarily, in some embodiments, the mapping relationship between the image and the posture may be as shown in Table 1, wherein the mapping relationship between the image and the posture may include a corresponding relationship between the image and the posture of the shooting module at the same moment.

[0083] Table 1

[0084] image Shooting module posture Image A Pose data a Image B Pose data b Image C Pose data c …… …… ImageN Pose data n

[0085] Furthermore, in the embodiments of the present application, Figure 2 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 2 ,like Figure 2 As shown, the image processing method may further include the following steps:

[0086] Step 105: When the color richness is greater than the color richness threshold, determine the image processing parameters corresponding to the first image.

[0087] In an embodiment of the present application, after determining the color richness of the first image, the color richness and the color richness threshold may be compared. If the color richness is greater than the color richness threshold, the image processing parameters corresponding to the first image may be further determined.

[0088] It can be understood that in an embodiment of the present application, after comparing the color richness and the color richness threshold, if the comparison result is that the color richness is greater than the color richness threshold, then it can be considered that the color richness of the first image is sufficient, and therefore the corresponding image processing parameters can be selected for determination.

[0089] Further, in an embodiment of the present application, assuming that the image processing parameters include white balance parameters, then when determining the image processing parameters corresponding to the first image, you can choose to use the gray world white balance algorithm to calculate the white balance parameters, where the white balance parameters of the three channels R, G, and B can be represented by three coefficients, for example, Kr, Kg, and Kb, and the white balance parameters of the three channels R, G, and B can represent the adjustment coefficients of the three channels R, G, and B of the first image.

[0090] Among them, the Gray World Algorithm is a commonly used Automatic White Balance (AWB) technology, which is based on the gray world assumption, that is, for an image with a lot of color changes, the average values ​​of its R, G, and B channels tend to the same gray value K. The algorithm adjusts the R, G, and B channels of the image to eliminate the influence of ambient light on the image color, so that the image can still maintain a relatively realistic color under different lighting conditions.

[0091] Exemplarily, in some embodiments, for the gray world white balance algorithm, the average values ​​of the three channels R, G, and B can be calculated first, and recorded as Ravg, Gavg, and Bavg respectively; at the same time, the gray value K is determined, where K can be a preset fixed value or obtained by calculating the average value of the three channels, for example, K = (Ravg + Gavg + Bavg) / 3; then, according to the gray value K and the average value of the three channels, the gain coefficients (white balance parameters) Kr, Kg, and Kb of the three channels R, G, and B are calculated respectively, that is, the corresponding image processing parameters are determined, for example, the gain coefficients can be calculated by Kr = K / Ravg, Kg = K / Gavg, and Kb = K / Bavg.

[0092] Step 106: Perform image processing on the first image according to image processing parameters corresponding to the first image to obtain a processed image.

[0093] In an embodiment of the present application, if the color richness is greater than the color richness threshold, after determining the image processing parameters corresponding to the first image, the first image can be further processed according to the image processing parameters corresponding to the first image to obtain a processed image.

[0094] Furthermore, in an embodiment of the present application, based on the determined image processing parameters, a variety of methods can be used to adjust and process the first image to obtain a corresponding processed image.

[0095] Exemplarily, in some embodiments, assuming that the image processing parameters include white balance parameters Kr, Kg, and Kb, the corresponding image processing method is white balance processing. Accordingly, when performing white balance processing on the first image according to the white balance parameters, it is possible to choose to multiply the value of the color channel corresponding to each pixel of the first image by the corresponding white balance parameter to obtain a new value, which is the process of adjusting the white balance of the image. For example, the R, G, and B values ​​of each pixel in the image are adjusted using the calculated gain coefficient (white balance parameters Kr, Kg, and Kb). The adjusted pixel values ​​are: Rnew = R×Kr, Gnew = G×Kg, and Bnew = B×Kb.

[0096] Furthermore, in the embodiments of the present application, Figure 3 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 3 ,like Figure 3 As shown, the image processing method may further include the following steps:

[0097] Step 107 : when the number of white point pixels is greater than the number threshold, determine the image processing parameters corresponding to the first image according to the image data of the white point pixels.

[0098] In an embodiment of the present application, after determining the number of white point pixels of the first image, the number of white point pixels and a number threshold can be compared. If the number of white point pixels is greater than the number threshold, the image processing parameters corresponding to the first image can be determined based on the image data of the white point pixels.

[0099] It can be understood that in the embodiments of the present application, after comparing the number of white point pixels and the number threshold, if the comparison result is that the number of white point pixels is greater than the number threshold, then it can be considered that there are white point pixels in the first image, and the white point pixels can be used to determine the image processing parameters corresponding to the first image.

[0100] Further, in an embodiment of the present application, assuming that the image processing parameters include white balance parameters, then when determining the image processing parameters corresponding to the first image based on the image data of the white point pixels, you can choose to first calculate the average values ​​of the three channels of all white point pixels based on the pixel values ​​of the three channels R, G, and B of the white point pixels, that is, calculate the average values ​​of the three channels R, G, and B, respectively recorded as Ravg, Gavg, and Bavg, and then use the average values ​​of the three channels to calculate the gain coefficients (white balance parameters) Kr, Kg, and Kb of the three channels R, G, and B, respectively. For example, the gain coefficients can be calculated by Kr=Gavg / Ravg, Kg=1, and Kb=Gavg / Bavg.

[0101] Step 108: Perform image processing on the first image according to image processing parameters corresponding to the first image to obtain a processed image.

[0102] In an embodiment of the present application, if the number of white spot pixels is greater than a quantity threshold, after determining the image processing parameters corresponding to the first image based on the image data of the white spot pixels, the first image can be further processed according to the image processing parameters corresponding to the first image to obtain a processed image.

[0103] Furthermore, in an embodiment of the present application, based on the determined image processing parameters, a variety of methods can be used to adjust and process the first image to obtain a corresponding processed image.

[0104] Exemplarily, in some embodiments, assuming that the image processing parameters include white balance parameters Kr, Kg, and Kb, the corresponding image processing method is white balance processing. Accordingly, when performing white balance processing on the first image according to the white balance parameters, it is possible to choose to multiply the value of the color channel corresponding to each pixel of the first image by the corresponding white balance parameter to obtain a new value, which is the process of adjusting the white balance of the image. For example, the R, G, and B values ​​of each pixel in the image are adjusted using the calculated gain coefficient (white balance parameters Kr, Kg, and Kb). The adjusted pixel values ​​are: Rnew = R×Kr, Gnew = G×Kg, and Bnew = B×Kb.

[0105] It can be seen that after determining the color richness or the number of white point pixels, if the color richness is greater than the color richness threshold, or the number of white point pixels is greater than the number threshold, then the image data of the first image can be directly used to further determine the image processing parameters corresponding to the first image. The order of the comparison process of the color richness and the color richness threshold and the process of the number of white point pixels being greater than the number threshold is not specifically limited.

[0106] For example, in some embodiments, the color richness may be compared with the color richness threshold, and if the color richness is greater than the color richness threshold, the image data of the first image may be directly used to further determine the image processing parameters corresponding to the first image. Otherwise, if the color richness is less than or equal to the color richness threshold, the number of white spot pixels may be compared with the number threshold.

[0107] For example, in some embodiments, the number of white point pixels may be compared with a number threshold, and if the number of white point pixels is greater than the number threshold, the image data of the first image may be directly used to further determine the image processing parameters corresponding to the first image. Otherwise, if the number of white point pixels is less than or equal to the number threshold, the color richness may be compared with the color richness threshold.

[0108] Step 103: determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image.

[0109] In an embodiment of the present application, after acquiring the posture data corresponding to the first image, a reference image corresponding to the first image can be further determined in the second image according to the posture data corresponding to the first image.

[0110] It is understood that in the embodiment of the present application, the second image is other images in the video image except the first image. The application does not specifically limit the number of the second images, and the second image can be all images in the video image except the first image, or can be part of the images in the video image except the first image.

[0111] Furthermore, in an embodiment of the present application, when determining a reference image corresponding to a first image in a second image based on the posture data corresponding to the first image, a candidate image can be first determined in the second image; wherein the candidate image includes an image for which image processing parameters have been determined; then, based on the posture data corresponding to the first image and the posture data corresponding to the candidate image, a similarity parameter between the first image and the candidate image is determined; finally, the reference image can be determined in the candidate image based on the similarity parameter.

[0112] It can be understood that in an embodiment of the present application, when determining a reference image, an image with known image processing parameters can be first selected from the second image as a candidate image, and then a reference image close to the posture of the shooting module that captured the first image can be determined from the candidate images based on the posture data corresponding to the first image.

[0113] Furthermore, in an embodiment of the present application, the similarity parameter between the first image and the candidate image can be calculated based on the pose data corresponding to the first image and the pose data corresponding to the candidate image, wherein the closer the poses of the first image and the candidate image are, the higher the similarity between the two.

[0114] It is understood that in the embodiments of the present application, the number of candidate images and reference images is not specifically limited in the present application. For example, the candidate image may be one or more frames of a video image, and the reference image may be one or more frames of a video image.

[0115] Furthermore, in an embodiment of the present application, the pose data corresponding to the candidate image can be determined by a pre-constructed mapping relationship between the image and the pose, that is, through the mapping relationship between the image and the pose, the pose data of the shooting module when the reference image is acquired through the shooting module can be determined.

[0116] Exemplarily, in some embodiments, it is assumed that the video image includes six frames of images A, B, C, D, E, and F, image A is the first image to be processed, and images B, C, D, E, and F are the second images, wherein the image processing parameters of images B, C, and D have been determined. Therefore, images B, C, and D can be used as candidate images, and then the similarity parameters between image A and images B, C, and D are calculated based on the posture data of each image, which are 80%, 90%, and 85%, respectively. Finally, it can be determined that the reference images corresponding to image A are images C and D.

[0117] Further, in an embodiment of the present application, when determining a reference image corresponding to a first image in a second image based on the posture data corresponding to the first image, the approximation parameter between the first image and the second image can be first determined based on the posture data corresponding to the first image and the posture data corresponding to the second image; then, a candidate image can be determined in the second image based on the approximation parameter; finally, a reference image can be determined in the candidate images; wherein the reference image includes an image for which image processing parameters have been determined.

[0118] It can be understood that in an embodiment of the present application, when determining a reference image, it is possible to first determine a candidate image in the second image that is close to the posture of the shooting module that captured the first image based on the posture data corresponding to the first image, and then select an image with known image processing parameters from the candidate images as the final reference image.

[0119] Furthermore, in an embodiment of the present application, the approximation parameter between the first image and the second image can be calculated based on the pose data corresponding to the first image and the pose data corresponding to the second image, wherein the closer the poses of the first image and the second image are, the higher the approximation between the two images is.

[0120] It is understood that in the embodiments of the present application, the number of candidate images and reference images is not specifically limited in the present application. For example, the candidate image may be one or more frames of a video image, and the reference image may be one or more frames of a video image.

[0121] Furthermore, in an embodiment of the present application, the posture data corresponding to the second image can be determined by a pre-constructed mapping relationship between the image and the posture, that is, through the mapping relationship between the image and the posture, the posture data of the shooting module when the second image is acquired by the shooting module can be determined.

[0122] For example, in some embodiments, assuming that the video image includes six frames of images A, B, C, D, E, and F, image A is the first image to be processed, and images B, C, D, E, and F are the second images. The similarity parameters between image A and images B, C, D, E, and F can be calculated based on the posture data of each image, which are 80%, 90%, 85%, 92%, and 83% respectively. Then, it can be determined that the candidate images corresponding to image A are images C, D, E, and F. Among them, the image processing parameters of images B, C, and D have been determined, so it can be finally determined that the reference images corresponding to image A are images C and D.

[0123] Furthermore, in an embodiment of the present application, the similarity parameter between two different image frames can determine the degree of similarity of the states of the shooting modules when the two image frames are acquired, that is, the larger the similarity parameter, the closer the states of the shooting modules when the two image frames are acquired.

[0124] Furthermore, in an embodiment of the present application, when determining similarity parameters between different images, the image posture data may be used in combination with the weight value for calculation, wherein the image posture data may include the image position data and posture data.

[0125] Exemplarily, in some embodiments, the position data and posture data of an image can be represented in the form of vectors, for example, using a translation vector T_vec = (tx, ty, tz) to represent the position data, and using a rotation vector R_vec = (rx, ry, rz) to represent the posture data, and calculating the Euler distance between the position data of different images, and calculating the Euler distance between the posture data of different images, and then combining the weight value and the calculated Euler distance to determine the similarity parameters between different images.

[0126] Exemplarily, in some embodiments, the similarity parameter Similar between the first image and other images (candidate images or second images) can be calculated by the following formula:

[0127] Simi lar=K1×di st(T_vec0, T_vec1)+K2× di st(R_vec0, R_vec1) (1)

[0128] Where, T_vec0 and R_vec0 are the position data and attitude data of the first image, T_vec1 and R_vec1 are the position data and attitude data of other images, K1 and K2 are adjustable weight values, which are the weights for balancing the similarity of position and attitude. The dist() function means calculating the Euler distance between two vectors.

[0129] Step 104: Perform image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0130] In an embodiment of the present application, after determining a reference image corresponding to the first image in the second image based on the posture data corresponding to the first image, the first image can be further processed based on image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0131] It can be understood that in the embodiments of the present application, for images whose image processing parameters cannot be directly calculated, the corresponding reference image can be determined in combination with the posture data, and then the parameters for image processing of the image can be determined based on the image processing parameters of the reference image, and the image can be processed using the parameters to obtain the corresponding processed image.

[0132] Furthermore, in an embodiment of the present application, the image processing parameters may include white balance parameters. When performing image processing on a first image based on the image processing parameters of a reference image to obtain a processed image corresponding to the first image, it is possible to choose to directly perform white balance processing on the first image according to the white balance parameters of the reference image to obtain the processed image.

[0133] It is understandable that in the embodiment of the present application, after determining one or more reference images corresponding to the first image, the white balance parameters of the reference images may be directly used as the white balance parameters of the first image to perform white balance processing on the first image. For example, the white balance parameters of any one of the reference images may be selected to perform white balance processing on the first image, or the balance parameters of the reference image having the largest similarity parameter with the first image may be used to perform white balance processing on the first image.

[0134] Furthermore, in an embodiment of the present application, the image processing parameters may include white balance parameters. When performing image processing on a first image based on the image processing parameters of a reference image to obtain a processed image corresponding to the first image, it is possible to first perform numerical calculations based on the white balance parameters of the reference image to determine the first white balance parameters; and then perform white balance processing on the first image according to the first white balance parameters to obtain a processed image.

[0135] It can be understood that in an embodiment of the present application, after determining one or more reference images corresponding to the first image, you can choose to first perform numerical operations on the white balance parameters of the reference image to obtain an operation result, and use the operation result as the first white balance parameter for performing white balance processing on the first image, and then use the first white balance parameter to perform white balance processing on the first image to obtain a processed image.

[0136] For example, in some embodiments, the present application does not specifically limit the type and method of numerical calculation. For example, a mean value calculation is performed on the white balance parameters of the reference image to obtain the first white balance parameter, or a mean value calculation is performed on the white balance parameters of the reference image to obtain the first white balance parameter.

[0137] To summarize, the image processing method proposed in the embodiment of the present application, for the image to be processed, if the color richness and / or the number of white point pixels of the image meet the conditions for accurately calculating the image processing parameters, then the image data of the image can be directly used to calculate the corresponding image processing parameters and perform image processing; if the color richness and the number of white point pixels of the image do not meet the conditions for accurately calculating the image processing parameters, that is, the accurate image processing parameters cannot be directly calculated, then the image posture data can be used to search for a reference image that is closest to the acquisition position and acquisition posture of the image, and perform image processing based on the image processing parameters of the reference image.

[0138] That is to say, for images whose image processing parameters cannot be directly calculated, the image processing method proposed in the embodiment of the present application can use the position and attitude approximation function to find the image that is closest to the position and attitude of the image when it was acquired, and use the same image processing parameters to perform image processing.

[0139] The embodiment of the present application provides an image processing method, which determines the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image; when the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the number threshold, obtain the posture data corresponding to the first image; determine the reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is another image in the video image other than the first image; perform image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image. That is to say, in the embodiment of the present application, it can be determined whether the image can calculate accurate image processing parameters according to the color richness and the number of white point pixels of the image. If not, it is necessary to refer to the image processing parameters of other images. For example, the posture data corresponding to the image is used to search for an image with a similar posture when shooting as a reference image, and then perform image processing based on the image processing parameters of the reference image. In this way, the accuracy of the image processing parameters can be ensured, and a more ideal image processing effect can be obtained, thereby improving the image processing performance.

[0140] Based on the above embodiments, another embodiment of the present application provides an image processing method. In order to obtain better image processing effects, the position and posture (pose data) of a shooting module (such as a camera) can be obtained through a SLAM algorithm, and the mapping relationship between the image and the pose can be pre-set. When performing image processing, for images for which accurate image processing parameters cannot be obtained, a reference image with a position and posture that is close but certain can be determined based on the pose data, and the image processing parameters of the reference image can be used for image processing, thereby improving the image processing effect.

[0141] Furthermore, in the embodiments of the present application, Figure 4 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 4 ,like Figure 4 As shown, the image processing method may include the following steps:

[0142] Step 201: When a shooting module shoots video images corresponding to a time sequence, position data of the shooting module is obtained according to the time sequence.

[0143] In the process of video image acquisition and recording through the shooting module, the position and posture of the shooting module can also be determined in real time through the sensor, that is, at the same time, there is a frame of image captured by shooting, and there is also the position and posture of the shooting module detected and calculated, that is, posture data.

[0144] Step 202: construct a mapping relationship between the image and the posture based on the video image and the posture data of the shooting module.

[0145] According to the image and posture data corresponding to each moment in the time series, the mapping relationship between the obtained image and posture can be constructed. That is, the mapping relationship between the image and posture includes the image corresponding to any moment in the time series and the posture data of the shooting module.

[0146] Step 203: Determine the color richness corresponding to the first image.

[0147] For any frame of the video image, such as the first image, the color richness corresponding to the first image can be determined, wherein the color richness can determine the color expression of the first image.

[0148] Step 204 , determine whether the color richness is greater than the color richness threshold, if so, execute step 205 , otherwise execute step 207 .

[0149] The color richness is compared with the color richness threshold. If the comparison result is that the color richness is less than or equal to the color richness threshold, it can be considered that the color richness of the first image is insufficient, the accuracy of the calculated image processing parameters is poor, and the number of white point pixels of the first image needs to be further determined. If the comparison result is that the color richness is greater than the color richness threshold, it can be considered that the color richness of the first image is sufficient, so the corresponding image processing parameters can be selected for determination.

[0150] Step 205: Determine image processing parameters corresponding to the first image.

[0151] Assuming that the image processing parameters include white balance parameters, if it is determined that the color richness of the first image is sufficient, then when determining the image processing parameters corresponding to the first image, you can choose to use the gray world white balance algorithm to calculate the white balance parameters, that is, determine the image processing parameters corresponding to the first image.

[0152] Step 206: Perform image processing on the first image according to the image processing parameters corresponding to the first image to obtain a processed image.

[0153] After the image processing parameters corresponding to the first image are determined, the first image may be further processed according to the image processing parameters corresponding to the first image to obtain a processed image.

[0154] Step 207: Determine the number of white point pixels corresponding to the first image.

[0155] Determine the number of white point pixels corresponding to the first image. The number of white point pixels can be used to determine the number of white point pixels in the first image.

[0156] Step 208 , determine whether the number of white spot pixels is greater than the number threshold, if yes, execute step 209 , otherwise execute step 210 .

[0157] The number of white point pixels is compared with the number threshold. If the comparison result is that the number of white point pixels is less than or equal to the number threshold, it can be considered that there are no white point pixels in the first image, and the image processing parameters cannot be determined using the white point pixels, that is, the image processing parameters obtained by calculation are less accurate. If the number of white point pixels is greater than the number threshold, it can be considered that there are white point pixels in the first image, and the image processing parameters corresponding to the first image can be determined using the white point pixels.

[0158] Step 209: Determine image processing parameters corresponding to the first image according to the image data of the white point pixels.

[0159] If it is determined that white point pixels exist in the first image, then image processing parameters corresponding to the first image may be determined based on image data of the white point pixels.

[0160] For example, first calculate the average values ​​of the three channels of all white point pixels based on the pixel values ​​of the three channels of R, G, and B of the white point pixels, and then use the average values ​​of the three channels to calculate the white balance parameters of the three channels of R, G, and B respectively, that is, determine the image processing parameters corresponding to the first image.

[0161] Step 210: Obtain posture data corresponding to the first image.

[0162] If the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the number threshold, then it can be considered that the first image does not meet the conditions for calculating the corresponding image processing parameters, that is, the accuracy of the calculated image processing parameters is poor. Therefore, it is determined not to calculate the image processing parameters corresponding to the first image, but to choose to refer to the image processing parameters of other images, and then it is necessary to further obtain the posture data corresponding to the first image.

[0163] Step 211: Determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image.

[0164] One implementation method is: when determining a reference image corresponding to a first image in a second image based on posture data corresponding to the first image, a candidate image can be first determined in the second image; wherein the candidate image includes an image for which image processing parameters have been determined; then, based on the posture data corresponding to the first image and the posture data corresponding to the candidate image, a similarity parameter between the first image and the candidate image is determined; finally, a reference image can be determined in the candidate image based on the similarity parameter.

[0165] Another implementation method is: when determining a reference image corresponding to a first image in a second image based on the posture data corresponding to the first image, the approximation parameter between the first image and the second image can be first determined based on the posture data corresponding to the first image and the posture data corresponding to the second image; then a candidate image can be determined in the second image based on the approximation parameter; finally, a reference image can be determined in the candidate image; wherein the reference image includes an image for which image processing parameters have been determined.

[0166] Step 212: Perform image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0167] One implementation is: when performing image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image, it is possible to directly perform image processing on the first image according to the image processing parameters of the reference image to obtain the processed image. For example, it is possible to directly perform white balance processing on the first image according to the white balance parameters of the reference image to obtain the processed image.

[0168] Another implementation is: when performing image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image, it is possible to first perform numerical calculations based on the image processing parameters of the reference image to determine the first image processing parameters; and then perform image processing on the first image according to the first image processing parameters to obtain a processed image. For example, it is possible to first perform numerical calculations based on the white balance parameters of the reference image to determine the first white balance parameters; and then perform white balance processing on the first image according to the first white balance parameters to obtain a processed image.

[0169] Furthermore, in the embodiments of the present application, Figure 5 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 5 ,like Figure 5 As shown, the image processing method may include the following steps:

[0170] Step 301: When a shooting module shoots video images corresponding to a time sequence, position data of the shooting module is obtained according to the time sequence.

[0171] Step 302: construct a mapping relationship between the image and the posture based on the video image and the posture data of the shooting module.

[0172] Step 303: Determine the number of white point pixels corresponding to the first image.

[0173] Step 304 , determine whether the number of white point pixels is greater than the number threshold, if yes, execute step 305 , otherwise execute step 307 .

[0174] The number of white point pixels is compared with the number threshold. If the comparison result is that the number of white point pixels is less than or equal to the number threshold, it can be considered that there are no white point pixels in the first image, and the image processing parameters cannot be determined using the white point pixels, that is, the accuracy of the calculated image processing parameters is poor, so it is necessary to further determine the color richness corresponding to the first image. If the number of white point pixels is greater than the number threshold, it can be considered that there are white point pixels in the first image, and the image processing parameters corresponding to the first image can be determined using the white point pixels.

[0175] Step 305: Determine image processing parameters corresponding to the first image according to the image data of the white point pixels.

[0176] Step 306: Perform image processing on the first image according to the image processing parameters corresponding to the first image to obtain a processed image.

[0177] Step 307: Determine the color richness corresponding to the first image.

[0178] Step 308 , determine whether the color richness is greater than the color richness threshold, if so, execute step 309 , otherwise execute step 310 .

[0179] The color richness is compared with the color richness threshold. If the comparison result is that the color richness is less than or equal to the color richness threshold, it can be considered that the color richness of the first image is insufficient, and the corresponding image processing parameters cannot be directly calculated, that is, the calculated image processing parameters have poor accuracy. If the comparison result is that the color richness is greater than the color richness threshold, it can be considered that the color richness of the first image is sufficient, so the corresponding image processing parameters can be selected for determination.

[0180] Step 309: Determine image processing parameters corresponding to the first image.

[0181] Step 310: Obtain posture data corresponding to the first image.

[0182] Step 311: Determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image.

[0183] Step 312: Perform image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0184] To summarize, the image processing method proposed in the embodiment of the present application, for the image to be processed, if the color richness and / or the number of white point pixels of the image meet the conditions for accurately calculating the image processing parameters, then the image data of the image can be directly used to calculate the corresponding image processing parameters and perform image processing; if the color richness and the number of white point pixels of the image do not meet the conditions for accurately calculating the image processing parameters, that is, the accurate image processing parameters cannot be directly calculated, then the image posture data can be selected to search for a reference image that is closest to the acquisition position and acquisition posture of the image, and perform image processing based on the image processing parameters of the reference image.

[0185] That is to say, for images whose image processing parameters cannot be directly calculated, the image processing method proposed in the embodiment of the present application can use the position and attitude approximation function to find the image that is closest to the position and attitude of the image when it was acquired, and use the same image processing parameters to perform image processing.

[0186] An embodiment of the present application provides an image processing method, which can determine whether the image can calculate accurate image processing parameters based on the color richness and the number of white point pixels of the image. If not, it is necessary to refer to the image processing parameters of other images. For example, the posture data corresponding to the image is used to search for an image with a similar posture when shooting as a reference image, and then the image processing is performed based on the image processing parameters of the reference image. In this way, the accuracy of the image processing parameters can be ensured, and a more ideal image processing effect can be obtained, thereby improving the image processing performance.

[0187] Based on the above embodiments, another embodiment of the present application provides an image processing method, which may include but is not limited to an image white balance processing method. In order to obtain a better automatic white balance (AWB) effect, the camera (shooting module) position and posture can be obtained through a SLAM algorithm, and a corresponding relationship can be established. When performing AWB, for images that are not sure of making correct AWB, AWB adjustment can be performed using image parameters that are close in position and posture but are sure, thereby improving the AWB effect.

[0188] Furthermore, in the embodiment of the present application, the implementation process of the image processing method mainly includes two parts, one part is image recording and posture detection, and the other part is the image AWB processing part.

[0189] For example, in some embodiments, Figure 6 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 6 ,like Figure 6 As shown, first turn on the camera for image acquisition (step 401). During the image recording process, run the SLAM algorithm while recording, calculate the current camera position and posture in real time and save (step 402). The origin of the camera position coordinates can be set at the position of the camera at the beginning, and the SLAM algorithm is not specifically limited in this application. For example, any SLAM algorithm such as ORB-SLAM, VINS-Fusion, etc. can be used, as long as the camera position and posture can be obtained. Finally, the corresponding relationship between the image and posture data can be constructed and saved (step 403).

[0190] For example, in some embodiments, Figure 7 Schematic diagram of the implementation process of the image processing method proposed in the embodiment of the present application Figure 7 ,like Figure 7As shown, during the image AWB processing, the color richness of each frame of the image can be calculated first (step 501), and then based on the color richness, it is determined whether the color richness of the corresponding image is sufficient. If it is sufficient, the corresponding AWB parameters can be calculated and the AWB processing can be performed (step 502). If it is insufficient, white point pixel detection can be further performed (step 503). If there are white point pixels in the image, the corresponding AWB parameters can be calculated and the AWB processing can be performed (step 504). If there are no white point pixels in the image, then the reference image with the closest corresponding position and posture is searched according to the corresponding posture data (step 505), and finally the AWB processing is performed using the AWB parameters of the searched reference image (step 506).

[0191] For example, in some embodiments, Figure 8 A schematic diagram of an implementation of the image processing method proposed in the embodiment of the present application is shown in FIG. Figure 8 As shown, the color richness of the image can be calculated and whether the richness is sufficient can be determined. For example, the scatter coordinates (R / G, B / G) corresponding to the pixel can be calculated based on the pixel values ​​of the three channels of R, G, and B. That is, the pixel values ​​of each pixel in the image corresponding to the three channels of R, G, and B can be used to calculate R / G and B / G, and the scatter coordinates (R / G, B / G) corresponding to the pixel can be formed according to the calculation results. All pixels in the image are traversed to obtain the scatter coordinates (R / G, B / G) corresponding to all pixels (step 601). Then, the mean of the scatter coordinates (R / G, B / G) corresponding to all pixels is calculated to obtain the mean coordinates (ave-x, ave-y) of all scatter coordinates (step 602). Next, for each pixel in the image, the Euler distance between the scatter coordinates of the pixel and the mean coordinates is calculated (step 603). Then, the standard deviation of the coordinate distance corresponding to all pixels can be calculated, and the calculated standard deviation d is defined as the color richness of the image (step 604). Finally, the color richness is compared with the color richness threshold. If the standard deviation d (color richness) is greater than the color richness threshold, it can be considered that the color richness of the image is sufficient, and a relatively accurate AWB parameter can be calculated.

[0192] For example, in some embodiments, Fig. 9 This is a schematic diagram of a scatter plot proposed in an embodiment of the present application, such as Fig. 9 As shown, a scatter plot generated based on image data of all pixels in the image has a horizontal coordinate of R / G and a vertical coordinate of B / G. The more dispersed the points in the scatter plot are, the richer the color of the image can be represented.

[0193] For example, in some embodiments, a grayscale world white balance algorithm is used to calculate white balance parameters for colorful images. Usually, the white balance parameters are represented by three coefficients, such as Kr, Kg, and Kb, which represent the adjustment coefficients of the image R, G, and B channels. The process of adjusting the image white balance is to multiply the value of the color channel corresponding to each pixel of the image by the corresponding coefficient to obtain a new value.

[0194] For example, in some embodiments, Fig.10 A schematic diagram of an implementation of the image processing method proposed in the embodiment of the present application is shown in FIG. Fig.10 As shown, pixels with R / G and B / G close to 1 and with larger image brightness are usually identified as white point pixels, so these pixels can be screened out using a threshold system. If the displayed number is sufficient (for example, the number of white point pixels is greater than the number threshold), it is considered that there are white point pixels in the image, and then the AWB parameter calculation based on the white point pixels can be performed. Among them, the overexposed pixels can be removed first to obtain the screened pixels (step 701). For example, the R / G and B / G corresponding to each pixel are calculated based on the pixel values ​​of the three channels of R, G, and B of each pixel in the image; then the preset color threshold (upper threshold th1 and lower threshold th2) and the color parameters of all pixels in the image are compared. If the comparison result of the color parameter of a pixel and the preset color threshold does not satisfy th2<R / G<th1 and th2<B / G<th1, then the pixel can be eliminated. Otherwise, if th2<R / G<th1 and th2<B / G<th1 are satisfied, then the pixel is retained to obtain the screened pixel. Next, based on the filtered pixels, the white point pixels and the number of white point pixels are determined (step 702). For example, the pixel values ​​of the three channels of R, G, and B of each filtered pixel are averaged to determine the pixel mean ave-rgb of the filtered pixel, and the pixel mean ave-rgb of the filtered pixel is compared with the preset pixel threshold th3. If ave-rgb>th3, then the filtered pixel can be determined as a white point pixel in the image, and each filtered pixel is traversed to determine the number of white point pixels in the image. If the number of white point pixels is greater than the number threshold, it can be considered that there are white point pixels in the image, so the average values ​​of the three channels of the white point pixels can be further calculated, that is, the average values ​​of the three channels of R, G, and B can be calculated, which are recorded as Ravg, Gavg, and Bavg respectively (step 703). Finally, the average values ​​of the three channels are used to calculate the gain coefficients (white balance parameters) Kr, Kg, and Kb of the R, G, and B channels respectively. For example, the gain coefficients can be calculated by Kr=Gavg / Ravg, Kg=1, and Kb=Gavg / Bavg (step 704).

[0195] To summarize, the image processing method proposed in the embodiment of the present application, for the image to be processed, if the color richness and / or the number of white point pixels of the image meet the conditions for accurately calculating the AWB parameters, then the image data of the image can be directly used to calculate the corresponding AWB parameters and perform AWB processing; if the color richness and the number of white point pixels of the image do not meet the conditions for accurately calculating the AWB parameters, that is, the accurate AWB parameters cannot be directly calculated, then the image posture data can be selected to search for a reference image that is closest to the acquisition position and acquisition posture of the image, and perform AWB processing based on the AWB parameters of the reference image.

[0196] That is to say, for images whose AWB parameters cannot be directly calculated, the image processing method proposed in the embodiment of the present application can use the position and attitude approximation function to find the image that is closest to the position and attitude of the image when it was acquired, and use the same AWB parameters to perform AWB processing.

[0197] An embodiment of the present application provides an image processing method, which can determine whether the image can calculate accurate image processing parameters based on the color richness and the number of white point pixels of the image. If not, it is necessary to refer to the image processing parameters of other images. For example, the posture data corresponding to the image is used to search for an image with a similar posture when shooting as a reference image, and then the image processing is performed based on the image processing parameters of the reference image. In this way, the accuracy of the image processing parameters can be ensured, and a more ideal image processing effect can be obtained, thereby improving the image processing performance.

[0198] Based on the above embodiment, in another embodiment of the present application, Fig.11 This is a schematic diagram of the structure of the image processing device proposed in the embodiment of the present application. Fig.11 As shown, the image processing device 110 proposed in the embodiment of the present application may include:

[0199] A determination unit 1101 is used to determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image;

[0200] An acquisition unit 1102 is used to acquire the posture data corresponding to the first image when the color richness is less than or equal to the color richness threshold and the number of white point pixels is less than or equal to the number threshold;

[0201] The determining unit 1101 is further configured to determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image;

[0202] The acquiring unit 1102 further performs image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0203] In the embodiments of the present application, further, Fig.12 This is a schematic diagram of the structure of the computer device proposed in the embodiment of the present application, such as Fig.12 As shown, the computer device 120 proposed in the embodiment of the present application may include a processor 1201 , a memory 1202 , a communication interface 1203 , and a bus 1204 for connecting the processor 1201 , the memory 1202 , and the communication interface 1203 .

[0204] In the embodiment of the present application, the above-mentioned processor 1201 can be at least one of an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processing Device, DSPD), a programmable logic device (ProgRAMmable Logic Device, PLD), a field programmable gate array (Field ProgRAMmable Gate Array, FPGA), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and the embodiment of the present application is not specifically limited. The computer device 120 can also include a memory 1202, which can be connected to the processor 1201, wherein the memory 1202 is used to store executable program code, the program code includes computer operation instructions, and the memory 1202 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0205] In the embodiment of the present application, the bus 1204 is used to connect the communication interface 1203, the processor 1201 and the memory 1202, as well as the mutual communication between these devices.

[0206] In practical applications, the memory 1202 may be a volatile memory, such as a random access memory (RAM); or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or a combination of the above types of memories, and provide instructions and data to the processor 1201.

[0207] Furthermore, in an embodiment of the present application, the processor 1201 is used to determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image; when the color richness is less than or equal to a color richness threshold, and the number of white point pixels is less than or equal to a number threshold, obtain the posture data corresponding to the first image; determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image; and perform image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0208] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0209] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0210] An embodiment of the present application provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the image processing method described above is implemented.

[0211] Specifically, the program instructions corresponding to an image processing method in this embodiment may be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to an image processing method in the storage medium are read or executed by an electronic device, the following steps are included:

[0212] Determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image;

[0213] When the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the number threshold, acquiring the posture data corresponding to the first image;

[0214] Determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image;

[0215] The first image is processed based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

[0216] The embodiment of the present application also provides a computer program product.

[0217] In some embodiments, the computer program product may include a computer program or instructions.

[0218] In some embodiments, the computer program product can be applied to the computer device in the embodiments of the present application, and the computer program instructions enable the computer to execute the corresponding processes implemented by the computer device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0219] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0220] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0221] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0222] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0223] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. An image processing method, It is characterized in that The method comprises: Determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image; When the color richness is less than or equal to the color richness threshold, and the number of white point pixels is less than or equal to the number threshold, acquiring the posture data corresponding to the first image; Determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image; The first image is processed based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

2. The method according to claim 1, It is characterized in that The method further comprises: When the video images corresponding to the time sequence are captured by the shooting module, the position and posture data of the shooting module are acquired according to the time sequence; Based on the video image and the posture data of the shooting module, a mapping relationship between the image and the posture is constructed; wherein the mapping relationship between the image and the posture includes the image corresponding to any moment in the time series and the posture data of the shooting module; The obtaining of the posture data corresponding to the first image includes: Based on the mapping relationship between the image and the posture, the posture data corresponding to the first image is obtained.

3. The method according to claim 2, It is characterized in that The determining, in the second image according to the posture data corresponding to the first image, a reference image corresponding to the first image, comprises: determining a candidate image in the second image; wherein the candidate image includes an image for which the image processing parameters have been determined; Determining a similarity parameter between the first image and the candidate image according to the pose data corresponding to the first image and the pose data corresponding to the candidate image; The reference image is determined among the candidate images according to the similarity parameter.

4. The method according to claim 2, It is characterized in that The determining, in the second image according to the posture data corresponding to the first image, a reference image corresponding to the first image, comprises: Determining a similarity parameter between the first image and the second image according to the pose data corresponding to the first image and the pose data corresponding to the second image; determining a candidate image in the second image according to the proximity parameter; The reference image is determined among the candidate images; wherein the reference image includes an image for which the image processing parameters have been determined.

5. The method according to any one of claims 1 to 4, It is characterized in that The image processing parameters include white balance parameters, and performing image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image includes: White balance processing is performed on the first image according to the white balance parameters of the reference image to obtain the processed image.

6. The method according to any one of claims 1 to 4, It is characterized in that The image processing parameters include white balance parameters, and performing image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image includes: Performing numerical calculation based on the white balance parameter of the reference image to determine a first white balance parameter; Performing white balance processing on the first image according to the first white balance parameter to obtain the processed image.

7. The method according to claim 1, It is characterized in that The determining the color richness corresponding to the first image includes: For each pixel in the first image, determining the scattered point coordinates corresponding to the pixel according to the image data of the pixel; Determining mean coordinates corresponding to the first image based on scattered point coordinates of all pixels in the first image; Determine the coordinate distance corresponding to each pixel based on the mean coordinate and the scattered point coordinates of each pixel; Determining the color richness based on the coordinate distance corresponding to each pixel; The method further comprises: In a case where the color richness is greater than the color richness threshold, determining an image processing parameter corresponding to the first image; The first image is processed according to the image processing parameters corresponding to the first image to obtain the processed image.

8. The method according to claim 1, It is characterized in that The determining the number of white point pixels corresponding to the first image includes: For each pixel in the first image, determining a color parameter corresponding to the pixel according to the image data of the pixel; Based on a preset color threshold and the color parameters of all pixels in the first image, performing pixel screening on the first image to obtain screened pixels; Determine a pixel mean of the filtered pixels according to the image data of the filtered pixels, and determine white spot pixels and the number of the white spot pixels in the first image based on the pixel mean of the filtered pixels and a preset pixel threshold; The method further comprises: When the number of white point pixels is greater than the number threshold, determining image processing parameters corresponding to the first image according to the image data of the white point pixels; The first image is processed according to image processing parameters corresponding to the first image to obtain the processed image.

9. An image processing device, It is characterized in that The image processing device comprises: A determination unit, configured to determine the color richness corresponding to a first image and the number of white point pixels corresponding to the first image; wherein the first image is any image in a video image; An acquisition unit, configured to acquire the posture data corresponding to the first image when the color richness is less than or equal to a color richness threshold and the number of white point pixels is less than or equal to a number threshold; The determination unit is further configured to determine a reference image corresponding to the first image in the second image according to the posture data corresponding to the first image; wherein the second image is an image other than the first image in the video image; The acquisition unit is further configured to perform image processing on the first image based on the image processing parameters of the reference image to obtain a processed image corresponding to the first image.

10. A computer device, It is characterized in that The computer device includes a processor and a memory storing instructions executable by the processor. When the instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a program stored thereon, It is characterized in that When the program is executed by a processor, the method described in any one of claims 1 to 8 is implemented.

Images