Imaging Device and Method Applicable to Multiple Scenarios Based on a Lightweight Optical Imaging System

Abstract

The present invention discloses a multi-scene imaging method and device based on a lightweight optical imaging system. The method classifies reference images to determine the scene type of each reference image, and displays the reference images contained in each scene type on a display. The lightweight optical system is used to capture the reference images on the display. The resulting blurred images are subjected to degradation, denoising, vignetting correction, and channel-by-channel chromatic aberration correction. The blurred images are then registered with the corresponding reference images to form sample pairs. For each scene type, the sample pairs corresponding to the scene type are used to optimize the parameters of a generative adversarial network, and the parameter-optimized generator is used as a restoration model. During application, the actual scene images captured by the lightweight optical system are classified, and then quality restoration is performed using the restoration model corresponding to the scene type to obtain a quality-restored image. This method can solve the problems of high blur and high off-axis chromatic aberration that are prone to occur in lightweight optical systems.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an imaging device and method applicable to multiple scenarios based on a lightweight optical imaging system. Background Art

[0002] Vision is the most intuitive way for people to understand and change the world, and technological advances have greatly expanded its capabilities. The pursuit of high-quality imaging has led to rapid development of related technologies. Generally speaking, image quality is improved from two perspectives: optimizing the optical system and enhancing the capabilities of subsequent image processing algorithms. The former has become highly mature after decades of development, but it often comes at the cost of increased system complexity, which is a significant challenge for today's commonly used portable devices. For example, in mobile phone camera modules, the complexity of the optical system results in excessively large modules, occupying a significant portion of the phone's space and protruding from the rear panel, significantly impacting both aesthetics and portability.

[0003] With the development of modern optical technology, the emergence of diffractive optical elements (DOEs) has made lightweight optical systems possible. DOEs have a spacious and flexible design space, good off-axis imaging behavior, and a thin and flat appearance. Whether using DOEs alone or in combination with refractive and diffractive elements, they are very effective for compact optical systems. However, DOEs are prone to severe chromatic aberration in wide-spectrum conditions, affecting the visual experience. In addition, the original imaging quality of extremely lightweight optical systems is still significantly different from that of mature multi-element optical systems, causing the design to deviate from the original intention of high-quality imaging. Fortunately, the emergence of neural networks with extremely strong image restoration capabilities has strongly promoted the practical application of lightweight optical systems.

[0004] In recent years, deep learning has become a hot field, and convolutional neural networks (CNNs), in particular, have seen rapid development. Generative adversarial networks (GANs) have shown outstanding performance in generating high-quality images, with numerous popular architectures proposed and proven effective. GANs typically consist of two modules: a generator and a discriminator. The generator is responsible for producing realistic images, while the discriminator attempts to distinguish between the generator's images and real images. During training, the two compete with each other, ultimately using the generator to produce high-quality images. Initially, the generator uses random values as initial iteration conditions, which poses a significant challenge in generating stable generation results.

[0005] However, regardless of the network architecture, under the premise of supervised learning, the restoration effect is highly dependent on the dataset. Common public datasets contain a wide variety of images, covering most common scenes in life. Networks trained on such datasets often perform relatively mediocrely. The interweaving of various potentially conflicting features makes it difficult for the network to spontaneously determine which category the actual scene belongs to, resulting in suboptimal restoration results.

[0006] In daily life, there is a great demand for photography of specific scenes, such as night scenes, portraits, flowers and plants. Their brightness and color distribution vary greatly, and people's orientation preferences are also different. It is extremely difficult to obtain satisfactory results in all three listed situations through a network, which will inevitably make the network structure very complex. Summary of the Invention

[0007] In view of the above, the purpose of the present invention is to provide an imaging device and method suitable for multiple scenes based on a lightweight optical imaging system, so as to overcome the defect that artifacts are prone to appear in images of specific scenes, and to solve the problem that lightweight optical imaging systems are prone to high blur and high off-axis chromatic aberration.

[0008] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0009] In a first aspect, an embodiment provides an imaging method applicable to multiple scenarios based on a lightweight optical imaging system, comprising the following steps:

[0010] Classifying the reference images to determine the scene type of each reference image, and displaying the reference images included in each scene type on a display;

[0011] A reference image on a display is captured using a lightweight optical system. The resulting blurred image is denoised, vignetting corrected, and corrected for channel-by-channel chromatic aberration. The resulting blurred image is then registered with the corresponding reference image to construct a sample pair.

[0012] For each scene type, we use sample pairs corresponding to the scene type to optimize the parameters of the generative adversarial network, and use the parameter-optimized generator as the recovery model.

[0013] During application, after scene classification is performed on the actual scene image captured by the lightweight optical system, quality restoration is performed using a restoration model corresponding to the scene type to which the actual scene image belongs, so as to obtain a quality restored image.

[0014] In one embodiment, a scene classifier is used to perform scene classification on the reference image and the actual scene image; the scene classifier performs scene classification based on core features of different scenes or scene labels of the reference image.

[0015] In one embodiment, image channel-wise chromatic aberration correction is implemented based on the image homography transformation principle.

[0016] In one embodiment, the image channel-wise chromatic aberration correction based on the image homography transformation principle includes:

[0017] According to the spatial correspondence between the RGB domain checkerboard blurred image and the standard checkerboard reference image, the homography matrix of each channel is constructed. The homography matrix is used to transform the images of each channel in the RGB domain of the blurred image into the relative spatial positions of the reference image. The relative spatial positions corresponding to each channel are then merged into an RGB three-channel image, thereby eliminating the offset of the position coordinates of light of different wavelengths caused by chromatic aberration and realizing channel-by-channel chromatic aberration correction.

[0018] In one embodiment, the process of registering the blurred image with the corresponding reference image and constructing a sample pair includes:

[0019] The blurred image is corrected for chromatic aberration in each channel to achieve registration between the blurred image and the reference image. The blurred image is then cropped according to the size of the reference image and the cropped blurred image is concatenated with the reference image to form a sample pair.

[0020] In one embodiment, the lightweight optical system includes a lightweight single-piece lens, or a lens group formed by multiple lightweight single-piece lenses.

[0021] In one embodiment, after optimizing the parameters of the generative adversarial network, the generator parameters and generator structure obtained by optimizing the sample pairs contained in each scene type are used as a branch to process images of each scene type, namely, a restoration model.

[0022] In a second aspect, an embodiment provides an imaging device applicable to multiple scenes based on a lightweight optical imaging system, comprising a lightweight optical system, a scene classifier, and multiple restoration models, wherein the restoration models are constructed using the imaging method described in the first aspect;

[0023] The lightweight optical system is used to capture images of actual scenes to obtain images of the actual scenes;

[0024] The scene classifier is used to perform scene classification on the actual scene image to determine the scene type of the actual scene image;

[0025] The restoration model is used to restore the quality of an input actual scene image of the same scene type to obtain a quality restored image.

[0026] In a third aspect, an embodiment provides an imaging method applicable to multiple scenarios based on a lightweight optical imaging system. The imaging method uses the imaging device described in the second aspect, and the imaging method includes the following steps:

[0027] Use a lightweight optical system to capture images of actual scenes to obtain images of actual scenes;

[0028] Using a scene classifier to perform scene classification on the actual scene image to determine the scene type of the actual scene image;

[0029] A restoration model corresponding to the scene type is selected according to the scene type to restore the quality of the input actual scene image to obtain a quality restored image.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] Based on the generative adversarial network, a restoration model adapted to each type of scene is constructed. On this basis, image restoration is performed by adapting the restoration model to various scene images collected by the lightweight optical imaging system to obtain high-quality images, taking into account both the high quality of imaging and the lightweight of the optical system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 is a flowchart of an imaging method applicable to multiple scenarios based on a lightweight optical imaging system provided in an embodiment;

[0034] Figure 2 is a block flow diagram of an imaging method applicable to multiple scenarios based on a lightweight optical imaging system provided in an embodiment;

[0035] Figure 3 2 is a schematic diagram of the structure of the generative adversarial network provided in the embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0037] Figure 1 is a flowchart of an imaging method applicable to multiple scenarios based on a lightweight optical imaging system provided in an embodiment; Figure 2 FIG. 1 is a block flow chart of an imaging method applicable to multiple scenarios based on a lightweight optical imaging system provided in an embodiment. Figure 1 and Figure 2 As shown, the embodiment provides an imaging method applicable to multiple scenarios based on a lightweight optical imaging system, including the following steps:

[0038] Step 1: Classify the scenes of the reference images to determine the scene type of each reference image, and display the reference images included in each scene type on the display.

[0039] In this embodiment, different scenes have different core features. Based on the different characteristics of the core feature imaging, scenes can be classified. Scene types include but are not limited to human scenes, natural scenes, and architectural scenes. Currently, based on lighting conditions, scene types can also be divided into normal-light scenes and low-light scenes. Normal-light scenes refer to scenes captured under normal lighting conditions, while low-light scenes refer to scenes in which the details of the scenery cannot be fully captured due to uneven or insufficient lighting during the shooting process.

[0040] In one embodiment, when constructing sample data, a scene classifier is designed for specific scenes based on the restoration scenario requirements. This scene classifier is used to classify reference image datasets containing different scenes, resulting in reference image datasets of different scene types. The scene classifier can be used to classify scenes based on their core features or based on existing scene labels in the dataset. The scene-classified reference images are displayed on a calibrated, high-quality display to facilitate the construction of blurred images corresponding to the reference images.

[0041] In an embodiment, when the reference image data set in the RGB domain is classified into normal-light scenes and low-light scenes, the principle of the scene classifier is to count the ratio of the number of dark pixels in the reference image to the total number of pixels. Low-light scene refers to a scene in which the lighting is uneven or insufficient. After being imaged by the lens on the sensor, it is processed by the ISP and appears as a lower grayscale value in the RGB channel image. When the grayscale value of a pixel is lower than the threshold value obtained by evaluating the actual scene, the pixel is considered to be a dark pixel. The number of dark pixels is counted and the ratio of the dark pixels to the total number of pixels is calculated. When the ratio is lower than the ratio threshold, the reference image is considered to belong to a low-light scene, otherwise it is a normal-light scene. The reference image belonging to the low-light scene is used as a training label and is marked as low-light GT, and the reference image belonging to the normal-light scene is used as a training label and is marked as normal-light GT.

[0042] The clear RGB domain dataset uses Adobe5K by default. In theory, any image with the same depth and similar size can be used.

[0043] Step 2: Use a lightweight optical system to capture the reference image on the display. After denoising, vignetting correction, and channel-by-channel chromatic aberration correction, the resulting blurred image is aligned with the corresponding reference image and a sample pair is constructed.

[0044] In the embodiments, a lightweight optical system refers to an optical system that uses fewer lenses, is smaller in size, and is lighter than conventional optical systems that achieve similar functions. This system can utilize a single lightweight lens or a lens assembly consisting of multiple lightweight single-lens lenses. The lightweight lens is a single Fresnel lens with a nearly spatially invariant PSF, whose front surface is aspheric and whose rear surface is a Fresnel surface. Obviously, this Fresnel lens is not the only lightweight lens implemented in the present invention and is provided as an example.

[0045] In an embodiment, a reference image of each scene type is displayed on a calibrated high-quality display, and the display is photographed using a lightweight optical system equipped with a lightweight lens to automatically acquire batches of RGB three-channel blurred images.

[0046] After obtaining the blurred image, denoising, vignetting correction and channel-by-channel chromatic aberration correction are performed on the blurred image to obtain a blurred image with inconspicuous chromatic aberration. The model image is matched, cropped and spliced with the corresponding reference image to obtain sample pairs of corresponding scene types for training the generative adversarial network.

[0047] Due to the various chromatic aberrations inherent in the lens in the lightweight optical system, light of different colors (wavelengths) propagates in space in different paths, and their relative spatial positions will shift, which is manifested as the relative position of the overlapping color information in the clear RGB image changes after imaging by the lens, resulting in the presence of color stripes in the blurred image. Thanks to the fixed shooting position of the camera and screen of the lightweight optical system, the acquired blurred image has a fixed spatial transformation relationship with the reference real image. Therefore, in the embodiment, channel-by-channel chromatic aberration correction is performed based on the principle of image homography transformation. Specifically, the homography matrix of each channel is constructed according to the spatial correspondence between the RGB domain checkerboard blurred image and the standard checkerboard reference image, and the homography matrix is used to transform the images of each channel of the RGB domain of the blurred image into the relative spatial position of the reference image, and then the relative spatial positions corresponding to each channel are merged into an RGB three-channel image, thereby eliminating the offset of the position coordinates of light of different wavelengths caused by chromatic aberration, and realizing channel-by-channel chromatic aberration correction.

[0048] In the embodiment, the imaging quality at the edge of the image field of view is significantly different from that at the center field of view. The imaging quality at the edge of the captured blurred image field of view is poor, and vignetting is obvious. The edge portion is cropped and aligned and spliced with the reference image to obtain blurred-real paired sample pairs to reduce the burden of network training.

[0049] Step 3: For each scene type, use the sample pairs corresponding to the scene type to optimize the parameters of the generative adversarial network, and use the parameter-optimized generator as the recovery model.

[0050] In this embodiment, the network structure used for training includes, but is not limited to, a GAN network structure model. This model is flexible and variable, depending on the imaging requirements. When using a GAN network, the GAN network parameters are optimized using sample pairs corresponding to each scene type. The generator parameters and generator structure obtained by optimizing the sample pairs included in each scene type serve as a branch for processing images of each scene type, thus forming the restoration model.

[0051] For normal-light scenes and low-light scenes, the sample pairs corresponding to the two scenes are input into the initial GAN network for training. The specific GAN network structure parameters are shown in Figure 3 During training, you can adjust the various loss functions, or the proportions of their subcomponents, as well as the sizes of hyperparameters such as Epoch and Batch Size, to achieve the desired results. In this embodiment, two GAN network generators, one for normal-light scenes and one for low-light scenes, are used as restoration models to construct a comprehensive restoration model.

[0052] In this embodiment, based on sample pairs from different scenarios as training samples, a restoration model that can achieve the best restoration effect in a specific scenario is trained as a network branch of the comprehensive restoration model. The branch network of multiple scenarios constructs a comprehensive restoration model suitable for multi-scenario fuzzy image restoration.

[0053] Step 4: When applied, after classifying the actual scene image captured by the lightweight optical system, quality restoration is performed using a restoration model corresponding to the scene type to which the actual scene image belongs, to obtain a quality restored image.

[0054] In the embodiment, a lightweight lens in a lightweight optical system is used to shoot an actual scene, and the blurred actual scene image obtained by shooting is identified as belonging to a scene by a scene classifier, a scene label is affixed, and the restoration model corresponding to the label is input for restoration to obtain a high-quality restoration result image.

[0055] In this embodiment, a lightweight lens is used to capture blurred images of normal-light and low-light scenes. After distortion correction, the blurred images are input into the scene classifier designed in step 1 to perform scene classification. If the blurred image belongs to a low-light scene, the corresponding sub-network molecule for the low-light scene, i.e., the corresponding recovery model (low-light model), is called to restore the blurred image. Conversely, if the blurred image belongs to a normal-light scene, the corresponding sub-network molecule for the normal-light scene, i.e., the corresponding recovery model (normal-light model), is called to restore the blurred image. Experiments have confirmed that the network branch trained based on the scene-by-scene dataset has higher evaluation indicators and better visual effects for blurred images belonging to the network branch.

[0056] An embodiment also provides an imaging device suitable for multiple scenes based on a lightweight optical imaging system, including a lightweight optical system, a scene classifier, and multiple restoration models, wherein the restoration model is constructed through the above-mentioned imaging method; the lightweight optical system is used to capture images of actual scenes to obtain actual scene images; the scene classifier is used to perform scene classification on actual scene images to determine the scene type of the actual scene images; the restoration model is used to restore the quality of input actual scene images of the same scene type to obtain quality-restored images.

[0057] In the imaging device provided in this embodiment, the restoration model comprises a branch of a comprehensive restoration model comprising multiple molecular structures, each branch corresponding to restoring a blurred image of an actual scene of the same scene type. During application, the blurred image of the actual scene type is input into the restoration model corresponding to the scene type, and the restoration model calculates a high-quality restored image.

[0058] An embodiment provides an imaging method applicable to multiple scenes based on a lightweight optical imaging system. The imaging method adopts the above-mentioned imaging device and includes the following steps: using a lightweight optical system to capture an image of an actual scene to obtain an actual scene image; using a scene classifier to classify the actual scene image to determine the scene type of the actual scene image; selecting a restoration model corresponding to the scene type according to the scene type to restore the quality of the input actual scene image to obtain a quality-restored image.

[0059] The above-mentioned embodiments provide an imaging method and device applicable to multiple scenes based on a lightweight optical imaging system. A restoration model adapted to each type of scene is constructed based on a generative adversarial network. On this basis, image restoration is performed by adapting the restoration model to various scene images collected by the lightweight optical imaging system to obtain high-quality images, thereby taking into account both the high quality of imaging and the lightweight of the optical system.

[0060] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-scene imaging method based on a lightweight optical imaging system, characterized in that: The following steps are involved: Classifying the reference images to determine the scene type of each reference image, and displaying the reference images included in each scene type on a display; A reference image on a display is captured using a lightweight optical system. After the obtained blurred image is subjected to denoising, vignetting correction, and channel-by-channel chromatic aberration correction, the obtained blurred image is aligned with the corresponding reference image to construct a sample pair. The channel-by-channel chromatic aberration correction of the image is achieved based on the principle of image homography transformation, including: constructing a homography matrix for each channel based on the spatial correspondence between the RGB domain checkerboard blurred image and the standard checkerboard reference image, using the homography matrix to transform the images of each channel in the RGB domain of the blurred image into the relative spatial positions of the reference image, and then merging the relative spatial positions corresponding to each channel into an RGB three-channel image, thereby eliminating the offset of the position coordinates of light of different wavelengths caused by chromatic aberration and achieving channel-by-channel chromatic aberration correction. The process of aligning the blurred image with the corresponding reference image and constructing the sample pair includes: performing channel-by-channel chromatic aberration correction on the blurred image to achieve alignment with the reference image, then cropping the blurred image according to the size of the reference image, and splicing the cropped blurred image with the reference image to form a sample pair. For each scene type, the sample pairs corresponding to the scene type are used to optimize the parameters of the generative adversarial network. The generator parameters and generator structure obtained by optimizing the sample pairs contained in each scene type are used as a branch to process images of each scene type, which is the restoration model. During application, after scene classification is performed on the actual scene image captured by the lightweight optical system, quality restoration is performed using a restoration model corresponding to the scene type to which the actual scene image belongs, so as to obtain a quality restored image.

2. The multi-scene imaging method based on a lightweight optical imaging system according to claim 1, characterized in that: A scene classifier is used to classify the reference image and the actual scene image; the scene classifier classifies the scene according to the core features of different scenes or the scene labels of the reference image.

3. The multi-scene imaging method based on a lightweight optical imaging system according to claim 1, characterized in that: The lightweight optical system includes a lightweight single-piece lens or a lens group formed by multiple lightweight single-piece lenses.

4. An imaging device applicable to multiple scenes based on a lightweight optical imaging system, characterized in that: The method comprises a lightweight optical system, a scene classifier, and a plurality of restoration models, wherein the restoration model is constructed by the imaging method according to any one of claims 1 to 3; The lightweight optical system is used to capture images of actual scenes to obtain images of the actual scenes; The scene classifier is used to perform scene classification on the actual scene image to determine the scene type of the actual scene image; The restoration model is used to restore the quality of an input actual scene image of the same scene type to obtain a quality restored image.

5. A multi-scene imaging method based on a lightweight optical imaging system, characterized in that: The imaging method uses the imaging device according to claim 4, and the imaging method includes the following steps: Use a lightweight optical system to capture images of actual scenes to obtain images of actual scenes; Using a scene classifier to perform scene classification on the actual scene image to determine the scene type of the actual scene image; A restoration model corresponding to the scene type is selected according to the scene type to restore the quality of the input actual scene image to obtain a quality restored image.

Images