Fish-eye image processing method and device, electronic equipment and storage medium

Abstract

The application discloses a fisheye image processing method, device, electronic equipment and storage medium. The fisheye image processing method comprises the following steps: selecting three sampling points from n sampling points in a fisheye image to obtain a sampling point group and a circle corresponding to each sampling point group; calculating the sum of distances from the n sampling points to the circle corresponding to any sampling point group as the error of the sampling point group; calculating a first cumulative value of the errors of the sampling point groups, and calculating a second cumulative value of the errors of the sampling point groups containing the sampling point p; calculating the weight coefficient of the p point according to the first cumulative value and the second cumulative value to construct a nonlinear optimization equation, and determining the effective imaging area of the fisheye camera according to the nonlinear optimization equation. The application can adapt to the determination of the effective imaging area of the fisheye image obtained under different scenes, has strong robustness, and ensures the accuracy of the determination of the effective imaging area of the fisheye image.

Description

Technical Field

[0001] This application relates to the field of fisheye image processing technology, and more specifically, to a fisheye image processing method, a fisheye image processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the process of distortion correction for fisheye cameras, it is essential to accurately determine the effective imaging area and optical center to ensure the correction effect. Before leaving the factory, accurate calibration is typically performed. However, during the use of a fisheye camera, collisions or other structural displacements can alter the effective imaging area, necessitating recalibration. However, post-factory calibration operates in a more complex environment than the production environment, leading to reduced accuracy in determining the effective imaging area based on images acquired by the fisheye camera after leaving the factory. Therefore, a robust method is urgently needed to accurately detect the effective imaging area. Summary of the Invention

[0003] This application provides a fisheye image processing method, a fisheye image processing apparatus, an electronic device, and a computer-readable storage medium.

[0004] The fisheye image processing method of this application includes extracting n sampling points from a fisheye image captured by a fisheye camera; arbitrarily selecting 3 sampling points from the n sampling points to obtain... A set of sampling points is defined, and the circle corresponding to each sampling point group is determined; for the... For any sampling point group in the n sampling point groups, calculate the sum of the distances from the n sampling points to the circle corresponding to the sampling point group, and use this sum as the error of the sampling point group; calculate the... The system calculates the first accumulated value of the error of each sampling point group, and for any sampling point p among the n sampling points, calculates the second accumulated value of the error of all sampling point groups containing point p; calculates the weight coefficient of point p based on the first accumulated value and the second accumulated value, the weight coefficient being negatively correlated with the second accumulated value; constructs a nonlinear optimization equation based on the weight coefficient, and determines the effective imaging area of ​​the fisheye camera based on the nonlinear optimization equation.

[0005] In some implementations, before extracting n sampling points from the fisheye image captured by the fisheye camera, the fisheye image processing method further includes: acquiring the original image captured by the fisheye camera; preprocessing the original image to obtain the fisheye image, wherein the preprocessing includes edge detection and image binarization.

[0006] In some embodiments, the pre-processing of the original image to obtain the fisheye image comprises down-sampling the original image.

[0007] In some embodiments, the calculating the weight coefficient of the p point according to the first accumulated value and the second accumulated value comprises calculating the weight coefficient of the p point according to the following formula: W P = 1 - ∑F P / ∑F, wherein the W P is the weight coefficient of the p point, the ∑F is the first accumulated value, and the ∑F P is the second accumulated value.

[0008] In some embodiments, the constructing the non-linear optimization equation according to the weight coefficient comprises constructing the non-linear optimization equation according to the following formula: wherein (x P , y P ) is the image coordinate of the p point, (x, y) is the image coordinate of the center of the effective imaging area, the W P is the weight coefficient of the p point, and the R is the radius of the effective imaging area.

[0009] In some embodiments, the determining the effective imaging area of the fisheye camera according to the non-linear optimization equation comprises: taking the partial derivative of x in the non-linear optimization equation to obtain a first equation; taking the partial derivative of y in the non-linear optimization equation to obtain a second equation; taking the partial derivative of R in the non-linear optimization equation to obtain a third equation; according to the first equation, the second equation, the third equation, and a preset gradient descent algorithm, taking values of x, y and R at a preset step to obtain a plurality of sets of values, and calculating a function value of the non-linear optimization equation corresponding to each set of values; and determining the effective imaging area according to the values corresponding to the minimum function value.

[0010] In some embodiments, after the constructing the non-linear optimization equation according to the weight coefficient and the determining the effective imaging area of the fisheye camera according to the non-linear optimization equation, the fisheye image processing method further comprises: extracting m sampling points whose distances from the center of the effective imaging area are within a preset distance range; reconstructing the non-linear optimization equation according to the weight coefficients corresponding to the m sampling points, and updating the effective imaging area according to the reconstructed non-linear optimization equation.

[0011] The fisheye image processing apparatus of this application includes a first extraction module, a first determination module, a first calculation module, a second calculation module, a third calculation module, and a second determination module. The first extraction module is used to extract n sampling points from a fisheye image captured by a fisheye camera; the first determination module is used to arbitrarily select 3 sampling points from the n sampling points to obtain... The first calculation module is used to calculate the number of sampling point groups and determine the circle corresponding to each sampling point group; For any sampling point group in the n sampling point groups, calculate the sum of the distances from the n sampling points to the circle corresponding to the sampling point group, as the error of the sampling point group; the second calculation module is used to calculate the... The first accumulated value of the error of each sampling point group is calculated, and for any sampling point p among the n sampling points, the second accumulated value of the error of all sampling point groups containing point p is calculated; the third calculation module is used to calculate the weight coefficient of point p based on the first accumulated value and the second accumulated value, the weight coefficient being negatively correlated with the second accumulated value; and the second determination module is used to construct a nonlinear optimization equation based on the weight coefficient, and determine the effective imaging area of ​​the fisheye camera based on the nonlinear optimization equation.

[0012] The electronic device according to embodiments of this application includes one or more processors, a memory, and one or more programs, wherein one or more of the programs are stored in the memory and executed by one or more of the processors, and the programs include instructions for performing the fisheye image processing method of any of the above embodiments.

[0013] The computer-readable storage medium of the embodiments of this application includes a computer program, which, when executed by a processor, causes the processor to perform the fisheye image processing method of any of the above embodiments.

[0014] The fisheye image processing method, fisheye image processing apparatus, electronic device, and computer-readable storage medium of this application extract n sampling points from a fisheye image, and then determine a group of any three sampling points from the n sampling points. There are n sampling point groups, and the circle corresponding to each sampling point group is determined; by calculating the sum of the distances from the n sampling points to the circle corresponding to each sampling point group, the error corresponding to each sampling point group can be determined; then, by determining... The first cumulative value of a group of sampling points and the second cumulative value of an error containing any sampling point p in the n sampling points are used to determine the weight coefficient of any sampling point p. It can be understood that the farther the distance between a sampling point and other sampling points, the greater the possibility of noise. Therefore, the greater the second cumulative value of the sampling point p, the farther the distance between the sampling point p and other sampling points, and the smaller the weight coefficient of the sampling point p. By determining the weight coefficient of any sampling point p in advance, the influence of the sampling point that may be noise in the fisheye image is reduced, and a nonlinear optimization equation for determining the effective imaging area of the fisheye image is constructed based on the weight coefficient of each sampling point. The optimal solution of the nonlinear optimization equation is calculated, and the effective imaging area of the fisheye image is accurately obtained. In this way, compared with accurately screening all target sampling points in different scenes that can be used to accurately calculate the effective imaging area, the present application does not need to screen the sampling points, but only needs to accurately calculate the weight coefficient of each sampling point in advance to reduce the influence of noise, and then construct a nonlinear optimization equation according to the weight coefficient of all sampling points. Not only can it adapt to the determination of the effective imaging area of the fisheye image captured in different scenes, but also has strong robustness and ensures the accuracy of the determination of the effective imaging area of the fisheye image.

[0015] Additional aspects and advantages of the embodiments of the present application will be in part apparent and in part pointed out hereinafter in the description of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the description of the embodiments, taken in conjunction with the following drawings in which:

[0017] Figure 1 is a flowchart of a fisheye image processing method according to some embodiments of the present application;

[0018] Figure 2 is a scene diagram of a fisheye image processing method according to some embodiments of the present application;

[0019] Figure 3 is a flowchart of a fisheye image processing method according to some embodiments of the present application;

[0020] Figure 4 is a flowchart of a fisheye image processing method according to some embodiments of the present application;

[0021] Figure 5 is a flowchart of a fisheye image processing method according to some embodiments of the present application;

[0022] Figure 6 is a flowchart of a fisheye image processing method according to some embodiments of the present application;

[0023] Figure 7 is a module schematic diagram of a fisheye image processing device according to some embodiments of the present application;

[0024] Figure 8 is a structural schematic diagram of an electronic device according to some embodiments of the present application;

[0025] Figure 9 is a connection state schematic diagram of a non-volatile computer readable storage medium and a processor according to some embodiments of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which the same or similar numerals or characters represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0027] Referring to Figure 1 , the embodiments of the present application provide a fisheye image processing method, the fisheye image processing method comprising:

[0028] Step 011: extracting n sampling points in a fisheye image captured by a fisheye camera;

[0029] Wherein, the fisheye image can be an original image collected by the fisheye camera, or the fisheye image is an image after preprocessing such as edge detection and image binarization on the original image collected by the fisheye camera.

[0030] The sampling point is a pixel in the fisheye image that meets the requirements, such as the sampling point being a pixel in the fisheye image whose pixel value is greater than a preset pixel value threshold; or the fisheye image is a binary image, the pixel value of the pixel includes 0 and 1, and the sampling point is a pixel in the fisheye image whose pixel value is 1;

[0031] After extracting n (n is an integer) sampling points in the fisheye image, in order to facilitate subsequent calculation, the image coordinates of the n sampling points in the fisheye image can be obtained.

[0032] Step 012: randomly selecting 3 sampling points from the n sampling points to obtain sampling point groups, and determining the circle corresponding to each sampling point group;

[0033] Wherein, the image coordinates of the sampling points are used to represent the positions of the sampling points in the fisheye image, such as establishing an image coordinate system with a preset position (such as the upper left corner, the upper right corner, the center, etc.) of the fisheye image as the coordinate origin, so as to determine the image coordinates of the sampling points in the image coordinate system according to the positions of the sampling points in the fisheye image.

[0034] The effective imaging area of the fisheye camera is circular, so when calculating the error of each sampling point, any 3 sampling points can be selected as a sampling point group, such as n sampling points, then (n! / [3!(n-3)!]) sampling point groups can be determined, such as 5 sampling points, that is, 5! / [3!(5-3)!]=10.

[0035] Each sampling point group includes 3 sampling points, and the 3 sampling points can determine a circle. For example, according to the image coordinates of the 3 sampling points of each sampling point group, the circle corresponding to each sampling point group is determined.

[0036] Step 013: for any sampling point group in the n sampling point groups, calculate the sum of the distances from the n sampling points to the circle corresponding to the sampling point group as the error of the sampling point group;

[0037] After determining the circle corresponding to each sampling point group, the sum of the distances from the n sampling points to the circle corresponding to each sampling point group can be calculated as the error of each sampling point group. It can be understood that the closer the circle determined by the sampling point group to the effective imaging area of the fisheye camera, the more sampling points on the circle determined by the sampling point group, and the smaller the distance between the sampling points outside the sampling point group and the circle determined by the sampling point group. Therefore, the sum of the distances from the n sampling points to the circle corresponding to the sampling point group can be taken as the error of the sampling point group. The greater the error of the sampling point group, the greater the deviation between the circle corresponding to the sampling point group and the effective imaging area of the fisheye camera, and the greater the possibility that the 3 sampling points in the sampling point group are noise points.

[0038] For example, the distance of a sampling point to the circle corresponding to the sampling point group can be the absolute value of the difference between the distance of the sampling point to the center of the circle corresponding to the sampling point group and the radius of the circle corresponding to the sampling point group. For example, the sampling points include 5, which are sampling point a1(x1, y1), sampling point a2(x2, y2), sampling point a3(x3, y3), sampling point a4(x4, y4) and sampling point a5(x5, y5), and the sampling point group A1(including sampling point a1, sampling point a2 and sampling point a3) determines a circle R1. At this time, the distances of sampling points a1 to a5 to circle R1 can be calculated, such as first calculating the distances of sampling points a1 to a5 to the center of circle R1, the distance of sampling point a1 to the center of circle R1(x A1 , y A1 ) is the distance of sampling point a2 to the center of circle R1(x A1 , y A1 ) is the distance of sampling point a3 to the center of circle R1(x A1 , y A1 ) is Calculate the center (x) of sampling point a4 and circle R1. A1 y A1 distance

[0039] Calculate the distance between sampling point a5 and the center of circle R1. Then, calculate the absolute value of the difference between the distance from each sampling point a1 to a5 to the center of circle R1 and the radius r of circle R1, so as to determine the distance from each sampling point a1 to a5 to circle R1. For example, calculate the absolute value of the difference between distance d1 and the radius r of circle R1, |d1-r|, the absolute value of the difference between distance d2 and the radius r of circle R1, |d2-r|, the absolute value of the difference between distance d3 and the radius r of circle R1, |d3-r|, the absolute value of the difference between distance d4 and the radius r of circle R1, |d4-r|, and the absolute value of the difference between distance d5 and the radius r of circle R1, |d5-r|. Then, use the sum of the distances from sampling point a1 to circle R1, |d1-r|, |d2-r|, |d3-r|, |d4-r|, and |d5-r|, as the error of sampling point group A1. The calculation of the error for other sampling point groups is similar to that for sampling point group A1, and will not be elaborated further here.

[0040] Step 014: Calculation The first accumulated value of the error of each sampling point group is calculated, and for any sampling point p among the n sampling points, the second accumulated value of the error of all sampling point groups containing point p is calculated.

[0041] To more accurately determine the probability that each sampling point is noise, we can first determine... The first accumulated value of the error of each sampling point group is used as the total error; for any sampling point p among the n sampling points, since point p may be located in multiple sampling point groups, the second accumulated value of the error of all sampling point groups containing point p can be calculated.

[0042] Step 015: Calculate the weight coefficient of point p based on the first and second accumulated values. The weight coefficient is negatively correlated with the second accumulated value.

[0043] After determining the first accumulated value and the second accumulated value of any sampling point p, the weight coefficient of the p point can be determined through the second accumulated value and the first accumulated value of the p point. For example, the greater the ratio of the second accumulated value and the first accumulated value of the p point, the greater the deviation of the sampling point group where the p point is located from the effective imaging area of the fisheye camera, and the greater the possibility that the p point is noise. For example, the greater the second accumulated value of the p point, the greater the possibility that the p point is noise, and the smaller the weight coefficient of the p point, that is, the weight coefficient of the sampling point is negatively correlated with the second accumulated value of the sampling point. The smaller the weight coefficient of the p point, the smaller the influence of the p point on the subsequent determination of the effective imaging area of the fisheye camera, thereby reducing the influence of the sampling point that may be noise on the accuracy of the determination of the effective imaging area of the fisheye camera.

[0044] For example, the weight coefficient of the p point is calculated by the following formula: W P = 1 - ∑F P / ∑F, where W P is the weight coefficient of the p point, ∑F is the first accumulated value, and ∑F P is the second accumulated value. In this way, the sum of the weight coefficients of all sampling points is 1, and the negative correlation between the second accumulated value of the sampling point and the weight coefficient is achieved.

[0045] Please refer to Figure 2 After leaving the factory, due to the dust on the edge of the lens of the fisheye camera during use, the need to add OSD content (such as Figure 2 “20XX-XX-XX”) to the photographed image after leaving the factory, uneven brightness of the shooting scene, and other situations, resulting in a large amount of noise inside and outside the boundary of the effective imaging area (such as Figure 2 OSD content and part of the circular area), the existence of burrs on the edge of the lens of the fisheye camera, resulting in the loss of the boundary (such as Figure 2 the gap on the circle), thereby affecting the accuracy of the determination of the effective imaging area. Therefore, for the above-mentioned scenarios, in order to ensure the accuracy of the determination of the effective imaging area, screening needs to be performed accordingly to remove noise, such as for OSD content, manually marking the OSD area for removal, adjusting the edge detection algorithm if there is dust on the edge of the lens, and the like. That is to say, for different scenarios, corresponding algorithms need to be set to ensure the accuracy of the effective imaging area of the fisheye image, and the algorithm cannot adapt to different scenarios, cannot guarantee the accuracy of the effective imaging area of the fisheye image in different scenarios, and has poor robustness.

[0046] In different scenes, after the fisheye image is acquired, the weight coefficient of each sampling point is determined based on the second accumulated value of the error of the sampling point group of each sampling point and the first accumulated value of the error of all sampling point groups, the greater the possibility that the sampling point is noise, the smaller the weight coefficient, thereby reducing the influence of noise and ensuring the accuracy of the effective imaging area of the fisheye image in subsequent calculation, which can adapt to fisheye images in different scenes and has strong robustness.

[0047] Step 016: constructing a nonlinear optimization equation according to the weight coefficient, and determining the effective imaging area of the fisheye camera according to the nonlinear optimization equation.

[0048] Specifically, after obtaining the weight coefficient of any sampling point p, the nonlinear optimization equation can be constructed according to the weight coefficient of the p point.

[0049] The effective imaging area of the fisheye camera is circular, so the center (i.e. optical center) and radius of the effective imaging area need to be determined to determine the effective imaging area of the fisheye camera. Therefore, when constructing the nonlinear optimization equation, the preset to-be-solved parameters in the nonlinear optimization equation include the image coordinates of the center of the effective imaging area and the radius. In this way, the effective imaging area can be determined according to the nonlinear optimization equation subsequently.

[0050] For example, the nonlinear optimization error equation can be constructed based on the image coordinates and weight coefficient of each sampling point and the preset to-be-solved parameters, so as to determine the effective imaging area through the nonlinear optimization error equation. For example, the nonlinear optimization equation is constructed according to the following formula: Wherein, (x P , y P ) is the image coordinates of the p point, (x, y) is the image coordinates of the center of the effective imaging area, W P is the weight coefficient of the p point, and R is the radius of the effective imaging area.

[0051] Wherein, the optimal solution of the nonlinear optimization equation is the minimum value of the function of the nonlinear optimization equation, and the value of the preset to-be-solved parameter is determined when the function value of the nonlinear optimization equation is the minimum value, i.e. the optimal solution of the nonlinear optimization equation.

[0052] After the nonlinear optimization equation is constructed, the nonlinear optimization equation can be solved to obtain the optimal solution of the nonlinear optimization equation. For example, the derivative of each to-be-solved parameter of the nonlinear optimization equation is calculated to determine the optimal solution of each to-be-solved parameter. After the optimal solution is determined, the optimal solution includes the radius and the image coordinates of the center of the effective imaging area, thereby determining the effective imaging area of the fisheye image.

[0053] The fisheye image processing method of the embodiments of the present application extracts n sampling points in the fisheye image, determines a group of any three sampling points in the n sampling points as a sampling point group, and determines a corresponding circle for each sampling point group; The error corresponding to each sampling point group is determined by calculating the sum of the distances from the n sampling points to the circle corresponding to each sampling point group. Then, the first cumulative value of the n sampling point groups and the second cumulative value of the error of any sampling point p in the n sampling points are determined to determine the weight coefficient of any sampling point p.

[0054] It can be understood that, since the noise points including OSD display are generally at the edges or corners of the effective imaging area (circular) of the fisheye camera, the sampling points at these edges or corners are obviously farther away from most of the sampling points in the image (such as the sampling points in the effective imaging area), and the farther the distance between a sampling point and other sampling points, the greater the possibility of being noise.

[0055] Therefore, the greater the second cumulative value of the sampling point p, the farther the distance between the sampling point p and other sampling points, and the smaller the weight coefficient of the sampling point p. By determining the weight coefficient of any sampling point p in advance, the influence of the sampling point that may be noise in the fisheye image is reduced, and a nonlinear optimization equation for determining the effective imaging area of the fisheye image is constructed based on the weight coefficient of each sampling point. The optimal solution of the nonlinear optimization equation is calculated, and the effective imaging area of the fisheye image is accurately obtained. In this way, compared with accurately screening the target sampling points in all sampling points that can be used to accurately calculate the effective imaging area in different scenes, the present application does not need to screen the sampling points, but only needs to accurately calculate the weight coefficient of each sampling point in advance to reduce the influence of noise, and then constructs the nonlinear optimization equation according to the weight coefficient of all sampling points. Not only can it adapt to the determination of the effective imaging area of the fisheye image captured in different scenes, but also has strong robustness and ensures the accuracy of the determination of the effective imaging area of the fisheye image.

[0056] Please refer to Figure 3 In some embodiments, step 016: constructing a nonlinear optimization equation according to the weight coefficient, comprises:

[0057] Step 0161: taking the partial derivative of x in the nonlinear optimization equation to obtain a first equation;

[0058] Step 0162: taking the partial derivative of y in the nonlinear optimization equation to obtain a second equation;

[0059] Step 0163: taking the partial derivative of R in the nonlinear optimization equation to obtain a third equation;

[0060] ​Step 0164: According to the first equation, the second equation, the third equation, and the preset gradient descent algorithm, the values of x, y and R are taken with a preset step size to obtain a plurality of sets of values, and the function value of the nonlinear optimization equation corresponding to each set of values is calculated.

[0061] Step 0165: The effective imaging area is determined according to the values corresponding to the minimum function value.

[0062] Specifically, after the nonlinear optimization equation is determined, the nonlinear optimization error equation is constructed as Then, the nonlinear optimization error equation can be solved to obtain the optimal solution of the nonlinear optimization error equation. Wherein, (x P , y P ) is the image coordinates of the p point, (x, y) is the image coordinates of the center of the effective imaging area, W P is the weight coefficient of the p point, and R is the radius of the effective imaging area.

[0063] The image coordinates of the center of the effective imaging area are (x, y), which can be determined by the partial derivative of x, y and R to determine the adjustment direction of x, y and R when the function value of the nonlinear optimization equation decreases, and then x, y and R are selected according to the adjustment direction of x, y and R by iteration to obtain a plurality of sets of x, y and R, so as to obtain a set of x, y and R that minimizes the function value of the nonlinear optimization equation as the optimal solution, and the effective imaging area of the fisheye camera can be determined according to the optimal solution.

[0064] For example, the first equation obtained by taking the partial derivative of x is used to determine the adjustment direction of x; the second equation obtained by taking the partial derivative of y is used to determine the adjustment direction of y; and the third equation obtained by taking the partial derivative of R is used to determine the adjustment direction of R.

[0065] At this time, according to the first equation, the second equation, the third equation, and the preset gradient descent algorithm (Gradient descent, GD), the values of x, y and R are taken with a preset step size to obtain a plurality of sets of values, such as a preset step size of 1, 2, etc. for x, a preset step size of 1, 2, etc. for y, and a preset step size of 1, 2, etc. for R. Each set of values includes x, y and R, and the function value of the nonlinear optimization equation corresponding to each set of values can be calculated. With continuous value taking, the function value of the nonlinear optimization equation continuously decreases, until the plurality of function values obtained by re-taking the value remain basically unchanged (such as the difference between the plurality of function values being less than a preset difference threshold), or the plurality of function values obtained by re-taking the value increase, so as to determine a set of values that minimizes the nonlinear optimization equation value as the optimal solution.

[0066] Referring to Figure 4 In some embodiments, before extracting the n sampling points in the fisheye image captured by the fisheye camera in step 011, the fisheye image processing method further comprises:

[0067] Step 017: obtaining an original image captured by the fisheye camera; and

[0068] Step 018: pre-processing the original image to obtain a fisheye image, the pre-processing comprising edge detection and image binarization.

[0069] Specifically, the fisheye camera will first generate an original image when shooting. After obtaining the original image captured by the fisheye camera, the original image can be pre-processed. It can be understood that the original image contains objects, characters and other objects in the scene. When determining the effective imaging area, it is generally necessary to determine the boundary of the original image and use the pixels of the boundary of the original image to determine the effective imaging area. Therefore, the original image needs to be pre-processed to filter a large number of pixels in the non-texture area, so as to retain as many pixels near the boundary as possible.

[0070] For example, the pre-processing of the original image can be edge detection (such as detecting the edge pixels in the original image based on a pre-set edge detection algorithm), and then image binarization processing, so as to obtain a fisheye image mainly containing edge pixels. After image binarization, the pixel value of the edge pixel is determined as 1, and the pixel value of the other pixel is determined as 0. At this time, the n sampling points in the pre-processed original image (i.e. the fisheye image) can be extracted.

[0071] It can be understood that in the fisheye image obtained by pre-processing the original image, in addition to the edge pixels of the boundary of the effective imaging area, there can also be background noise such as object edge pixels and OSD area pixels distributed inside and outside the boundary. These pixels all participate in the determination of the effective imaging area as subsequent sampling points, so as to adapt to the determination of the effective imaging area of the fisheye image obtained under any scene.

[0072] Referring to Figure 5 In some embodiments, step 018: pre-processing the original image to obtain a fisheye image, comprises:

[0073] Step 0181: down-sampling the original image;

[0074] Step 0182: pre-processing the down-sampled original image to obtain a fisheye image.

[0075] Specifically, after the original image is acquired, in order to reduce the calculation amount of subsequent effective imaging area determination, the original image can be down-sampled first. The down-sampling can reduce the number of sampling points. For example, after the original image is down-sampled according to a sampling coefficient k, the ratio of the size of the down-sampled original image to the size of the original image before down-sampling is 1 / (+1) 2 . For example, for an image of N*M, if the down-sampling coefficient is k, then in the original image, every k pixels in each row and each column are taken to form an image, thereby realizing down-sampling of the image, and the size of the down-sampled image becomes (N*M) / (+1) 2 . It should be explained that k+1 can be a common divisor of N and M, so as to ensure the accuracy of the size of the down-sampled image (i.e. (N*M) / (+1) 2 ).

[0076] It can be understood that when the size of the original image is too large, the number of sampling points in the original image after preprocessing will be too large. In order to reduce the calculation amount of calculating the effective imaging area, it is necessary to reduce the number of extracted sampling points. Therefore, the original image can be down-sampled. For example, if the fisheye image is an image of 1000*1000 and the down-sampling coefficient is 1, then in the original image, every 1 pixel in each row and each column is taken to form an image, which is used as the down-sampled original image. The down-sampled original image becomes 500*500. In this way, by down-sampling the original image, the number of pixels of the original image after down-sampling is reduced, thereby reducing the number of sampling points extracted from the fisheye image, so as to reduce the calculation amount of calculating the effective imaging area.

[0077] After the down-sampled original image is acquired, the down-sampled original image can be preprocessed (i.e. edge detection, image binarization, etc.), thereby obtaining the fisheye image.

[0078] The application finally obtains the effective imaging area of the fisheye camera, which is the effective imaging area of the fisheye image before down-sampling. After the fisheye image is down-sampled, the size of the fisheye image is different from that before down-sampling, and therefore the effective imaging area of the fisheye image is also different from that of the fisheye image before down-sampling. Therefore, the effective imaging area of the fisheye image can be restored according to the sampling coefficient to determine the effective imaging area of the fisheye camera. For example, the sampling coefficient is 1, the radius of the effective imaging area of the fisheye image is R, and the center of the effective imaging area is (x, y). It can be understood that the center of the effective imaging area is basically unchanged, and the radius is changed. The center of the effective imaging area of the fisheye camera can be determined as (x, y), and the radius of the effective imaging area of the fisheye camera is 2R, i.e., the radius of the effective imaging area of the fisheye camera = (sampling coefficient + 1) * radius of the effective imaging area of the fisheye image, so as to determine the effective imaging area of the fisheye camera.

[0079] Referring to Figure 6 In some embodiments, after the step 016 of constructing a nonlinear optimization equation according to the weight coefficient and determining the effective imaging area of the fisheye camera according to the nonlinear optimization equation, the fisheye image processing method further comprises:

[0080] Step 019: extracting m sampling points whose distance from the center of the effective imaging area is within a preset distance range;

[0081] Step 020: reconstructing the nonlinear optimization equation according to the weight coefficient corresponding to the m sampling points, and updating the effective imaging area according to the reconstructed nonlinear optimization equation.

[0082] Specifically, sampling points near the boundary of the effective imaging region of the fisheye image can be further filtered to correct the effective imaging region of the fisheye image. Sampling points near the boundary of the effective imaging region can be m sampling points (m being an integer) whose distance from the center of the effective imaging region of the fisheye image is within a preset distance range. For example, if the radius of the effective imaging region of the fisheye image is R, m sampling points whose distance from the center of the effective imaging region of the fisheye image is close to the radius R can be obtained, such as those with a preset distance range of (R±1), (R±2), etc. Thus, m sampling points near the boundary of the effective imaging region of the fisheye image can be extracted. Using the more accurate image coordinates and weight coefficients of these m sampling points, the nonlinear optimization equation can be reconstructed. The optimal solution of the reconstructed nonlinear optimization equation is then calculated, and the effective imaging region of the fisheye camera can be updated based on the newly determined optimal solution, thereby improving the accuracy of the finally determined effective imaging region of the fisheye camera. The method for calculating the optimal solution of the reconstructed nonlinear optimization equation is basically similar to the method for calculating the optimal solution of the nonlinear optimization equation, and will not be elaborated here.

[0083] Please see Figure 7 To facilitate better implementation of the fisheye image processing method of this application, this application also provides a fisheye image processing apparatus 10. The fisheye image processing apparatus 10 includes a first extraction module 11, a first determination module 12, a first calculation module 13, a second calculation module 14, a third calculation module 15, and a second determination module 16. The first extraction module 11 is used to extract n sampling points from a fisheye image captured by a fisheye camera; the first determination module 12 is used to arbitrarily select 3 sampling points from the n sampling points to obtain... The first calculation module 13 is used to calculate the number of sampling point groups and determine the circle corresponding to each sampling point group; For any sampling point group in the n sampling point groups, calculate the sum of the distances from the n sampling points to the circle corresponding to the sampling point group, which is used as the error of the sampling point group; the second calculation module 14 is used to calculate The first accumulated value of the error of the sampling point group is calculated, and for any sampling point p in the n sampling points, the second accumulated value of the error of all sampling point groups containing point p is calculated; the third calculation module 15 is used to calculate the weight coefficient of point p based on the first accumulated value and the second accumulated value, and the weight coefficient is negatively correlated with the second accumulated value; and the second determination module 16 is used to construct a nonlinear optimization equation based on the weight coefficient, and determine the effective imaging area of ​​the fisheye camera based on the nonlinear optimization equation.

[0084] The fisheye image processing apparatus 10 further includes an acquisition module 17 and a preprocessing module 18. The acquisition module 17 is used to acquire the original image captured by the fisheye camera. The preprocessing module 18 is used to preprocess the original image to obtain a fisheye image, and the preprocessing includes edge detection and image binarization.

[0085] The preprocessing module 18 is specifically further configured to down-sample the original image; and pre-process the down-sampled original image to obtain the fisheye image.

[0086] The third calculation module 15 is specifically configured to calculate the weight coefficient of the p point by the following formula: W P = 1 - ∑F P / ∑F, where W P is the weight coefficient of the p point, ∑F is the first accumulated value, and ∑F P is the second accumulated value.

[0087] The determination module 16 is specifically configured to construct the nonlinear optimization equation according to the following formula: where (x P , y P ) is the image coordinate of the p point, (x, y) is the image coordinate of the center of the effective imaging area, W P is the weight coefficient of the p point, and R is the radius of the effective imaging area.

[0088] The determination module 16 is specifically further configured to take the partial derivative of x in the nonlinear optimization equation to obtain a first equation; take the partial derivative of y in the nonlinear optimization equation to obtain a second equation; take the partial derivative of R in the nonlinear optimization equation to obtain a third equation; according to the first equation, the second equation, the third equation, and a preset gradient descent algorithm, take values of x, y and R at a preset step to obtain a plurality of sets of numerical values, and calculate a function value of the nonlinear optimization equation corresponding to each set of numerical values; and determine the effective imaging area according to the numerical values corresponding to the minimum function value.

[0089] The fisheye image processing apparatus 10 further includes a second extraction module 19 and an updating module 20. The second extraction module 19 is configured to extract m sampling points whose distances from the center of the effective imaging area are within a preset distance range; and the updating module 20 is configured to reconstruct the nonlinear optimization equation according to the weight coefficients corresponding to the m sampling points, and update the effective imaging area according to the reconstructed nonlinear optimization equation.

[0090] Referring to Figure 8 , the electronic device 100 of the embodiment includes one or more processors 20, a memory 30, and one or more programs 40, where the one or more programs 40 are stored in the memory 30 and executed by the one or more processors 20, and the program 40 includes instructions for executing the fisheye image processing method of any of the above embodiments.

[0091] The electronic device 100 can be a fisheye camera or a background server device, a mobile phone, a tablet computer, a notebook computer, a smart watch, etc. in communication with the fisheye camera. Figure 8As shown, the embodiments of the present application take the fish-eye camera as an example to illustrate the electronic device 100, and it can be understood that the specific form of the electronic device 100 is not limited to the fish-eye camera, that is, the fish-eye camera of the present application can determine the effective imaging area based on the fish-eye image after obtaining the fish-eye image, thereby facilitating the correction of the fish-eye camera at any time during use.

[0092] Please refer to Figure 9 The embodiments of the present application also provide a non-volatile computer readable storage medium 300, which stores a computer program 310, and the computer program 310 is executed by a processor 320 to implement the steps of the fish-eye image processing method of any one of the above embodiments. For brevity, details are not repeated here.

[0093] In the description of the present specification, the description of the terms "some embodiments", "in an example", "exemplarily", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0094] Any process or method descriptions in flow charts or described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for performing specific logic functions or steps in the process. The scope of preferred embodiments of the present application includes the additional implementation in which the functions can be performed in different order, in substantially simultaneous fashion, or in reverse order, according to the functions involved, as will be understood by those skilled in the art of the embodiments to which the present application belongs.

[0095] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A fisheye image processing method, characterized by, The fisheye image processing method comprises: extracting n sampling points in a fisheye image captured by a fisheye camera; Any 3 sampling points are selected from the n sampling points, to obtain a sampling point group, and a circle corresponding to each sampling point group is determined. Regarding the For any sampling point group in the n sampling point groups, calculate the sum of the distances from the n sampling points to the circle corresponding to the sampling point group, and use it as the error of the sampling point group; Calculate the The first accumulated value of the error of each sampling point group is calculated, and for any sampling point p among the n sampling points, the second accumulated value of the error of all sampling point groups containing point p is calculated; calculating a weight coefficient of the p point according to the first cumulative value and the second cumulative value, the weight coefficient being in a negative correlation with the second cumulative value; The nonlinear optimization equation is constructed according to the following formula: Wherein, (x P , y P ) is the image coordinate of the p point, (x, y) is the image coordinate of the center of the effective imaging area, the W P is the weight coefficient of the p point, and the R is the radius of the effective imaging area. determining an effective imaging area of the fisheye camera according to the nonlinear optimization equation.

2. The fisheye image processing method of claim 1, wherein, Before the extracting of the n sampling points in the fisheye image captured by the fisheye camera, the fisheye image processing method further comprises: obtaining an original image captured by the fisheye camera; preprocessing the original image to obtain the fisheye image, the preprocessing comprising edge detection and image binarization.

3. The fisheye image processing method of claim 2, wherein, The preprocessing of the original image to obtain the fisheye image comprises: down-sampling the original image; preprocessing the down-sampled original image to obtain the fisheye image.

4. The fisheye image processing method of claim 1, wherein, The calculating of the weight coefficient of the p point according to the first cumulative value and the second cumulative value comprises: The weight coefficient of the p point is calculated by the following formula: W P = 1 -∑F P / ∑F, wherein the W P is the weight coefficient of the p point, the∑F is a first cumulative value, and the∑F P is a second cumulative value.

5. The fisheye image processing method of claim 1, wherein, The determining of the effective imaging area of the fisheye camera according to the nonlinear optimization equation comprises: taking a partial derivative of x in the nonlinear optimization equation to obtain a first equation; taking a partial derivative of y in the nonlinear optimization equation to obtain a second equation; taking a partial derivative of R in the nonlinear optimization equation to obtain a third equation; according to the first equation, the second equation, the third equation and a preset gradient descent algorithm, taking values of x, y and R at a preset step to obtain a plurality of groups of values, and calculating a function value of the nonlinear optimization equation corresponding to each group of values; determining the effective imaging area according to the values corresponding to the minimum function value.

6. The fisheye image processing method of claim 1, wherein, After the constructing of the nonlinear optimization equation according to the weight coefficient and the determining of the effective imaging area of the fisheye camera according to the nonlinear optimization equation, the fisheye image processing method further comprises: extracting m sampling points whose distances from the center of the effective imaging area are within a preset distance range; reconstructing the nonlinear optimization equation according to the weight coefficients corresponding to the m sampling points, and updating the effective imaging area according to the reconstructed nonlinear optimization equation.

7. A fisheye image processing apparatus characterized by comprising: comprises: a first extraction module configured to extract n sampling points in a fisheye image captured by a fisheye camera; The first determining module is used to arbitrarily select 3 sampling points from the n sampling points to obtain... There are 10 groups of sampling points, and the circle corresponding to each group of sampling points is determined. a first calculation module, configured to calculate, for any one of the n sample point groups, a sum of distances from the n sample points to a circle corresponding to the sample point group as an error of the sample point group. a first calculation module, configured to calculate, for any one of the n sample point groups, a sum of distances from the n sample points to a circle corresponding to the sample point group as an error of the sample point group. a second calculation module, configured to calculate a first accumulated value of errors of the n groups of sampling points, and for any sampling point p in the n sampling points, calculate a second accumulated value of errors of all groups of sampling points containing the point p; a second calculation module, configured to calculate a first accumulated value of errors of the n groups of sampling points, and for any sampling point p in the n sampling points, calculate a second accumulated value of errors of all groups of sampling points containing the point p; a third calculation module configured to calculate a weight coefficient of the p point according to the first cumulative value and the second cumulative value, the weight coefficient being in a negative correlation with the second cumulative value; and A second determining module, configured to construct a nonlinear optimization equation according to a formula as follows: wherein (x P , y P ) is an image coordinate of the p point, (x, y) is an image coordinate of a center of an effective imaging area, W P is a weight coefficient of the p point, and R is a radius of the effective imaging area, and the effective imaging area of the fisheye camera is determined according to the nonlinear optimization equation.

8. An electronic device, comprising: comprises: one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs comprise instructions for executing the fisheye image processing method of any one of claims 1-6. 9.A non-volatile computer readable storage medium containing a computer program, which, when executed by a processor, causes the processor to execute the fisheye image processing method of any one of claims 1-6.

Images