Anti-halo interference fisheye image effective area detection method

Abstract

The invention discloses an anti-halo interference fisheye image effective area detection method, which performs modeling simulation on halo interference possibly existing in a fisheye image shooting process aiming at halo intensity, fuzzy degree, color, position and size factors, and constructs a fisheye image effective area detection model based on a deep neural network. The input of the model is a halo interference fisheye image, the output is the center coordinate and size of an effective area, and the anti-halo interference fisheye image effective area detection model comprises a fisheye image type identification module, an effective area identification module and an effective area parameter extraction module. The fisheye image effective area detection method provided by the invention has stronger robustness and higher detection precision for various halo interferences, can effectively solve the problems existing in the traditional method, and further improves the distortion parameter estimation precision and the distortion correction effect of the fisheye image.

Description

TECHNICAL FIELD

[0001] The present application relates to target detection technology, in particular to an effective area detection method for fish-eye image with anti-halo interference. BACKGROUND

[0002] The effective area of the fish-eye image refers to the pixel area containing specific scene information in the fish-eye image, and the distortion information of the fish-eye image is contained in the effective area. According to the distribution of the effective area of the fish-eye image, the fish-eye image can be divided into three types: circular fish-eye image, drum-shaped fish-eye image and full-frame fish-eye image. Among them, the effective area of the full-frame fish-eye image is the image itself, while the effective area of the circular fish-eye image and the drum-shaped fish-eye image is part of the image. During the shooting process, the fish-eye camera may be affected by the light and the optical characteristics of the lens, resulting in halo in the shot fish-eye image, that is, a circle of blurred light ring at the periphery of the effective area of the fish-eye image. In addition, the edge of the fish-eye image may include a black area without scene information. Both the halo and the black area belong to the invalid area in the fish-eye image. During the distortion correction process of the fish-eye image, the halo part may reduce the accuracy of the subsequent fish-eye image distortion parameter estimation and affect the distortion correction effect of the fish-eye image. Therefore, it is necessary to extract the effective area in the fish-eye image first. The traditional fish-eye image effective area extraction method is slow in calculation speed and is easily disturbed by different types of halo factors, which may further include part of the halo area in the effective area, resulting in low accuracy of the effective area detection, and the effective area detection needs to manually set the threshold for judgment. Through the analysis and modeling of the halo interference of the fish-eye image, and on this basis, the effective area is extracted by using the deep learning method, which can help to overcome the shortcomings of the traditional method and further improve the accuracy of the fish-eye image effective area extraction.

[0003] The fish-eye image effective area detection method proposed in the present application has stronger robustness and higher detection accuracy, and can effectively solve the problems existing in the traditional detection method, and further improve the distortion parameter estimation accuracy and distortion correction effect of the fish-eye image. SUMMARY

[0004] Therefore, it is necessary to provide an effective area detection method for fish-eye image with anti-halo interference, which is composed of an effective area identification module and an effective area parameter extraction module. The position information of the effective area is located during the process of completing the effective area identification, and the identification error and the positioning error are reduced through iterative training, so as to solve the problem that the existing technology cannot accurately extract the effective area of the fish-eye image with halo interference. The fish-eye image effective area detection method with anti-halo interference mainly includes the following steps:

[0005] Step one: model and simulate the halo interference factors, and construct the halo interference fisheye image simulation dataset. Each sample in the dataset contains a halo fisheye image, fisheye image class label and fisheye image effective area parameters. Among them, the fisheye image class includes three categories of circular fisheye image, drum-shaped fisheye image and full-frame fisheye image, and the fisheye image effective area parameters include the center point coordinates, width and height of the effective area. When constructing the dataset, first generate circular, drum-shaped and full-frame fisheye images through image cropping. Then analyze the halo interference factors in detail, and model and simulate the halo interference that may exist in the fisheye image shooting process according to the halo intensity, blur degree, color, position and size factors. Among them, only the halo interference factors are added to the circular and drum-shaped fisheye images to generate halo interference fisheye simulation images. Label the fisheye image class, effective area center point coordinates and size to complete the construction of the halo interference fisheye image simulation dataset. The dataset is mixed and randomly divided into training set, validation set and test set. The halo interference fisheye image simulation dataset contains training set, validation set and test set sample number division ratio of 6:2:2.

[0006] Step two: Construct an effective area detection model for fisheye images against vignetting interference, which includes a fisheye image type identification module, an effective area identification module, and an effective area parameter extraction module. The fisheye image type identification module first classifies the input fisheye image. If the input image is determined to be a full-frame fisheye image, the entire image is directly taken as the effective area, with the effective area center being the center of the input image and the effective area size being the size of the input image. If the input image is determined to be a circular or drum-shaped fisheye image, the effective area needs to be further detected using the effective area identification module and the effective area parameter extraction module. The effective area identification module distinguishes the effective area from the invalid area containing vignetting by extracting deep semantic information of the image data through multiple convolution and pooling operations. The effective area parameter extraction module locates the effective area, with the input being a fisheye image with vignetting interference and the output being the center coordinates, width, and height of the effective area. The effective area identification module and the effective area parameter extraction module share the same convolutional neural network, which mainly includes four modules: an input end, a backbone network, a neck structure, and a head structure, responsible for vignetting image preprocessing, feature extraction, feature fusion, and output detection information, respectively. The input end uses data augmentation by randomly scaling, cropping, and rearranging and then splicing to enrich data samples and improve the robustness of the effective area identification model. The backbone network is mainly composed of convolutional layers, batch normalization layers (BN), LeakyRelu, and residual blocks. The neck structure captures semantic features of strong fisheye images from top to bottom and then conveys strong positioning features from bottom to top to complete effective area positioning. The head structure completes the selection of target frames based on the loss function. The loss function for model training includes fisheye image category classification loss, effective area identification classification loss, positioning loss, and confidence loss. The classification loss uses cross-entropy as the loss function to constrain the model to correctly identify the effective area. The positioning loss is evaluated using an IoU-based loss calculation method, and the confidence loss also uses cross-entropy as the loss function.

[0007] Step three: Train the effective area detection model for fisheye images against vignetting interference using the dataset constructed in step one to obtain the trained effective area detection model for fisheye images against vignetting interference. The training process of the proposed model can be implemented using deep learning frameworks such as PyTorch, Tensorflow, or MXNet.

[0008] Step four: Use the trained effective area detection model in step three to classify and detect the effective area of the input fisheye image with vignetting interference, generating the center coordinates and size information of the effective area of the fisheye image with vignetting interference.

[0009] The present application has the beneficial effects that, compared with the traditional fisheye image effective area detection method, the method of the present application uses a deep neural network to extract the effective area of the light fisheye image, uses a fisheye image type recognition module to first classify the input fisheye image, and uses an effective area recognition module and an effective area parameter extraction module to fully mine the deep semantic information of the effective area and the light halo part, has better robustness to different types of light halo interference, and does not need to manually set a discrimination threshold, and quickly and accurately completes the fisheye image effective area detection. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a flowchart of the method of the present application;

[0011] Figure 2 is a schematic diagram of a fisheye image and light halo interference;

[0012] Figure 3 is a schematic diagram of different types of fisheye images, from left to right, they are circular, drum-shaped and full-frame fisheye images;

[0013] Figure 4 is the overall architecture of the fisheye image effective area detection model against light halo interference;

[0014] Figure 5 is the effect of effective area detection based on the method of the present application, and the Valid value marked in the figure is the confidence. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0016] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0017] As shown in Figure 1 , the anti-light halo interference fisheye image effective area detection method provided by the present embodiment mainly implements the following steps:

[0018] Step 1: Model and simulate the light halo interference factor, and construct the fisheye image simulation dataset of light halo interference. Figure 2Fish-eye image and halo interference schematic diagram. Each sample in the data set contains a fish-eye image with halo, fish-eye image class label and fish-eye image valid region parameters. Among them, the fish-eye image class includes three types of circular fish-eye image, drum-shaped fish-eye image and full-frame fish-eye image, and the fish-eye image valid region parameters include the center point coordinates, width and height of the valid region. Figure 3 are schematic diagrams of different types of fish-eye images. When constructing the data set, first generate circular, drum-shaped and full-frame fish-eye images by image cropping. Then, the halo interference factors are analyzed in detail, and the halo interference that may exist during the shooting of fish-eye images is modeled and simulated according to the factors of halo intensity, blur degree, color, position and size. Among them, only the halo interference factors are added to the circular and drum-shaped fish-eye images to generate fish-eye simulation images with halo interference. In order to label the fish-eye image class, the valid region center point coordinates and size, the fish-eye image simulation data set with halo interference is constructed. The data set is mixed and randomly divided into training set, validation set and test set. The fish-eye image simulation data set with halo interference contains training set, validation set and test set sample number division ratio of 6:2:2.

[0019] Step two: construct an anti-halo interference fish-eye image valid region detection model, which includes a fish-eye image type recognition module, a valid region recognition module and a valid region parameter extraction module. Figure 4is the overall architecture of the effective area detection model for fisheye images against halo interference. The fisheye image type recognition module first classifies the input fisheye image. If the input image is judged to be a full-frame fisheye image, the entire image is directly taken as the effective area, where the center of the effective area is the center of the input image, and the size of the effective area is the size of the input image. If the input image is judged to be a circular or drum-shaped fisheye image, the effective area needs to be further detected by the effective area recognition module and the effective area parameter extraction module. The effective area recognition module distinguishes the effective area from the invalid area containing halo by multiple convolution and pooling operations to extract deep semantic information of the image data. The effective area parameter extraction module locates the effective area, with the input being the fisheye image with halo interference and the output being the center coordinates, width and height of the effective area. The effective area recognition module and the effective area parameter extraction module share the same convolutional neural network, which mainly includes four modules of input end, backbone network, neck structure and head structure responsible for halo image preprocessing, feature extraction, feature fusion and output detection information respectively. The input end uses data augmentation by randomly scaling, cropping and rearranging and splicing to enrich the data samples and improve the robustness of the effective area recognition model. The backbone network is mainly composed of convolutional layers, batch normalization layers (BN), LeakyRelu and residual blocks. The neck structure captures the semantic features of strong fisheye images from top to bottom and then conveys the strong positioning features from bottom to top to complete the positioning of the effective area. The head structure completes the screening of the target frame based on the loss function. The loss function for model training includes fisheye image category classification loss, effective area recognition classification loss, positioning loss and confidence loss. Among them, the classification loss uses cross-entropy as the loss function to constrain the model to correctly identify the effective area. The positioning loss is evaluated by using the IoU-based loss calculation method, and the confidence loss also uses cross-entropy as the loss function. In the specific experiment process, the batch size is set to 32, the number of training rounds is set to 100, and the size of the input image is set to 640x640.

[0020] Step three: use the dataset constructed in step one to train the effective area detection model for fisheye images against halo interference, and obtain the trained effective area detection model for fisheye images against halo interference. The training process of the proposed model can be realized by using deep learning frameworks such as PyTorch, Tensorflow or MXNet. In the training process, the Adam optimizer is used, and the initial learning rate is set to 10 -4 . To ensure good recognition effect, the early stopping patience value is set to 100 to ensure sufficient training of the model.

[0021] Step four: use the effective area detection model trained in step three to classify and detect the effective area of the input halo interference fisheye image, and generate the center coordinates and size information of the effective area of the halo interference fisheye image. The test effect is as followsFigure 5 As shown from the effective area detection result, even in the case of halo interference, the proposed model can accurately detect the effective area in the fisheye image.

[0022] Specifically, the fisheye simulation image with halo interference is generated by the following steps:

[0023] A halo intensity function H(x, y) is established, and the expression is:

[0024] H(x, y) = I0·e -αd(x,y) ·PSF(x, y; sigma, lambda) + (x, y)

[0025] Where I0 is the initial intensity of the halo, which is related to the brightness of the light source, alpha is the light attenuation coefficient, which reflects the attenuation characteristics of the medium, d(x, y) is the distance between the current pixel point and the projection center of the light source, PSF(·) represents the point spread function based on Gaussian or Lambert projection, sigma is the blur degree parameter, lambda is the wavelength related dispersion parameter; (x, y) is a random noise term, simulating environmental random disturbance, x and y represent the horizontal and vertical coordinates of the current pixel point respectively;

[0026] The time dimension t is introduced, and a halo dynamic evolution model is established:

[0027] H(x, y, t) = H(x, y, t-1)·(1-beta) + I0(t)·e -αd(x,y,t) ·PSF(x, y, t)

[0028] Where beta is the halo attenuation rate, which simulates the change of halo caused by light source movement or light intensity fluctuation, I0(t), d(x, y, t), PSF(x, y, t) are the light source intensity, distance function and scattering function at time t respectively;

[0029] A conditional generative adversarial network is designed to automatically learn the characteristics of real halo, realize high-fidelity and diversified halo interference generation, the generator input halo parameter vector includes intensity, size, position and color, and the output is a fisheye image with halo; the discriminator distinguishes between real and generated images, and promotes the model to generate natural halo samples.

[0030] Further, composite interference is added to the circular and drum-shaped fisheye images, the composite interference includes: halo superposition scattering haze model, adding multiple real interference of heterogeneous light spot, noise, stain; edge halo and global light attenuation model are designed for full-frame fisheye image.

[0031] The fisheye image type recognition module is designed as a joint learning module for fisheye image type classification and halo interference degree estimation, realizing type recognition and halo interference degree quantitative evaluation, and the output is class probability And halo intensity estimation value

[0032] The joint loss function of the joint learning module is defined as:

[0033]

[0034] wherein, is the cross-entropy classification loss; is the halo intensity mean square error; are the effective region positioning loss and effective region confidence loss, respectively; λ * is the loss weight hyperparameter;

[0035] The halo attention module based on the self-attention mechanism is adopted in the joint learning module to dynamically adjust the feature map weight to suppress the interference of the halo region on the effective region detection. The attention weight calculation is A = σ (W2·ReLU (W1·F)), wherein F is the feature map, W1 and W2 are trainable weights, σ is the Sigmoid activation, the feature map weighted output is F' = A⊙F, and the halo region interference is removed.

[0036] Further, the spatial pyramid pooling structure is adopted in the joint learning module to extract multi-scale context information and improve the discrimination ability for different sizes of halos and effective regions. A special halo residual denoising subnetwork is designed, the halo is regarded as an image residual, and the residual information for effectively removing the halo interference is automatically learned through end-to-end training to assist in extracting pure effective region features.

[0037] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure

[0038] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

Claims

1. A method for detecting the effective region of a fisheye image against halo interference, characterized in that, The method comprises the following steps: Step 1: modeling and simulation of halo interference factors, and construction of a halo interference fisheye image simulation dataset; each sample in the dataset contains a halo fisheye image, a fisheye image category label and fisheye image effective area parameters, wherein the fisheye image category includes three categories of circular fisheye image, drum-shaped fisheye image and full-frame fisheye image, and the fisheye image effective area parameters include the center point coordinates, width and height of the effective area; Step 2: construction of an anti-halo interference fisheye image effective area detection model, including a fisheye image type identification module, an effective area identification module and an effective area parameter extraction module, the fisheye image type identification module first classifies the input fisheye image as a circular, drum-shaped or full-frame fisheye image, the effective area identification module distinguishes the effective area and the invalid area containing halo by multiple convolution and pooling operations to extract deep semantic information of the image data, and the effective area parameter extraction module positions the effective area, wherein the input is a halo interference fisheye image, and the output is the center point coordinates, width and height of the effective area; Step 3: training the anti-halo interference fisheye image effective area detection model using the dataset constructed in step 1 to obtain the trained anti-halo interference fisheye image effective area detection model; Step 4: using the effective area detection model trained in step 3 to detect the effective area of the input halo interference fisheye image to generate the effective area center coordinates and size information of the halo interference fisheye image.

2. The method of claim 1, wherein the method further comprises: In the construction of the dataset in step 1, first, circular, drum-shaped and full-frame fisheye images are generated by image cropping, then halo interference factors are analyzed in detail, and halo interference existing in the process of fisheye image shooting is modeled and simulated, wherein only halo interference factors are added to circular and drum-shaped fisheye images to generate halo interference fisheye simulation images, and the fisheye image category, effective area center point coordinates and size are labeled to complete the construction of the halo interference fisheye image simulation dataset.

3. The method of claim 2, wherein the method further comprises: determining a center of the fisheye image; and determining a radius of the fisheye image. The halo interference fisheye simulation image comprises the following steps: A halo intensity function H(x, y) is established, and the expression is: H(x, y) = I0e -αd(x,y) · PSF(x, y; σ, λ) + (x, y) Wherein I0 is the initial intensity of halo, which is related to the brightness of the light source, alpha is the light attenuation coefficient, which reflects the attenuation characteristics of the medium, d(x, y) is the distance between the current pixel point and the projection center of the light source, PSF(·) represents the point spread function based on Gaussian or Lambert projection, sigma is the blur degree parameter, and lambda is the wavelength related dispersion parameter; (x, y) is a random noise term, which simulates environmental random disturbance, and x and y represent the horizontal and vertical coordinates of the current pixel point, respectively; A time dimension t is introduced, and a halo dynamic evolution model is established: H(x, y, t) = H(x, y, t - 1) · (1 - β) + I0(t) · e -αd(x,y,t) · PSF(x, y, t) Wherein beta is a halo attenuation rate in [0, 1], which simulates the change of halo caused by light source movement or light intensity fluctuation, I0(t), d(x, y, t) and PSF(x, y, t) are the light source intensity, distance function and scattering function at time t, respectively. The design condition generative adversarial network is used to automatically learn real halo characteristics, realize high-fidelity and diversified halo interference generation, a generator input halo parameter vector includes intensity, size, position and color, and an output is a fisheye image with a halo; a discriminator distinguishes between real and generated images, and promotes the model to generate natural halo samples.

4. The method of claim 3, wherein the method further comprises: determining a center of the fisheye image; and determining a radius of the fisheye image. In the circular and drum-shaped fisheye images, composite interference is added, and the composite interference includes: halo superimposed scattering haze model, adding heterogeneous light spot, noise, stain multiple reality interference; an edge halo and global light attenuation model is designed for the full-frame fisheye image.

5. The method of claim 1, wherein the method further comprises: determining a center of the fisheye image; and determining a radius of the fisheye image. In step two, the fisheye image type recognition module first classifies the input fisheye image, if it is judged that the input image is a full-frame fisheye image, the whole image is directly taken as an effective area, wherein the effective area center is the input image center, and the effective area size is the input image size; if it is judged that the input image is a circular or drum-shaped fisheye image, the effective area needs to be further detected by using the effective area recognition module and the effective area parameter extraction module, the effective area recognition module and the effective area parameter extraction module share the same convolutional neural network, including an input end, a main network, a neck structure and a head structure, which are respectively responsible for halo image preprocessing, feature extraction, feature fusion and output detection information, the input end adopts data enhancement, which enriches the data samples through random scaling, cropping, arrangement and splicing, and improves the robustness of the effective area recognition model, the main network is mainly composed of a convolutional layer, a batch normalization layer, a LeakyRelu and a residual block, the neck structure captures the semantic features of the strong fisheye image from top to bottom, and then conveys the strong positioning features from bottom to top to complete the effective area positioning, and the head structure completes the screening of the target frame based on the loss function, and the loss function of the model training includes the fisheye image category classification loss, the effective area recognition classification loss, the positioning loss and the confidence loss, wherein the classification loss adopts cross entropy as the loss function to constrain the model to correctly recognize the effective area, the positioning loss adopts the loss calculation method based on IoU for evaluation, and the confidence loss also adopts cross entropy as the loss function.

6. The method of claim 5, wherein the method further comprises: The fisheye image type recognition module is designed as a joint learning module for fisheye image type classification and halo interference degree estimation, realizes type recognition and halo interference degree quantitative evaluation, and outputs a category probability and a halo intensity estimation value The joint loss function of the joint learning module is defined as: wherein, is a cross-entropy classification loss; is a halo intensity mean squared error; are respectively an effective region localization loss and an effective region confidence loss; λ * are loss weight hyperparameters; The joint learning module adopts a halo attention module based on a self-attention mechanism, dynamically adjusts the feature map weight to suppress the interference of the halo region on the effective region detection, and the attention weight calculation is A = σ (W2·ReLU (W1·F)), wherein F is a feature map, W1 and W2 are trainable weights, and σ is a Sigmoid activation, and the feature map weighted output is F ′ =A⊙F, and the halo region interference is removed.

7. The method of claim 6, wherein the method further comprises: determining a center of the fisheye image; and determining a radius of the fisheye image. In the joint learning module, a spatial pyramid pooling structure is used to extract multi-scale context information and improve the discrimination ability for different size halos and effective areas; a special halo residual denoising sub-network is designed, the halo is regarded as an image residual, and the residual information for effectively eliminating the halo interference is automatically learned through end-to-end training to assist in extracting pure effective area features.

8. The method of claim 1, wherein the method further comprises: determining a center of the fisheye image; and determining a radius of the fisheye image. In step one, the sample number division ratio of the training set, the validation set and the test set included in the fisheye image simulation dataset of the halo interference is 6:2:

2.

9. The method of claim 1, wherein the method further comprises: determining a center of the fisheye image; and determining a radius of the fisheye image. In step three, the training process of the fisheye image effective area detection model against halo interference is realized by using a PyTorch, Tensorflow or MXNet deep learning framework.

Images