A camera imaging optimization method and system based on complex lighting environment

Abstract

The application discloses a camera imaging optimization method and system based on a complex light environment, wherein an sRGB image with low brightness or overexposure is input into a U-Net neural network for inference calculation to obtain a linear Raw-RGB image before camera image signal processing; the Raw-RGB image is input into another U-Net neural network to obtain a pixel-by-pixel adjustment map, and the ambient light intensity is adjusted and corrected; subsequently, the Raw-RGB image after adjustment is subjected to inference of image signal processing parameters, including a white balance parameter, a color space conversion matrix and a gamma correction parameter, by using a neural network composed of a self-attention mechanism coding module; and finally, the Raw-RGB image is restored to an sRGB image with adjusted brightness through a simulated image signal processing process. The application avoids directly processing the nonlinear sRGB image output by the camera, adjusts the light intensity of the linear Raw-RGB image obtained through conversion, and finally restores the nonlinear sRGB image, thereby having good ability to restore low-brightness and overexposed images.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a camera imaging optimization method and system based on a complex light environment. BACKGROUND

[0002] When a camera is imaging, the strength of the ambient light directly determines the quality of the camera imaging. When the ambient light is weak, such as at night, the number of photons entering the photosensitive element of the camera is insufficient, resulting in a decrease in image brightness and accompanying noise; when the ambient light is too strong, such as directly facing a light source, the photosensitive element of the camera reaches its physical saturation point, resulting in excessively high image brightness and overexposure; when a strong light source is partially blocked in the shooting environment, the local brightness of the camera imaging is too low, and local overexposure occurs. Changes in the camera ambient light make it very challenging for the camera to shoot stable and clear images.

[0003] Foreign researchers have proposed a method of using a convolutional neural network model to input a low-brightness image, and then the model directly infers an image output after correcting the brightness. This method directly processes the non-linear sRGB image of the camera imaging, uses the fitting ability of the deep learning model for the non-linear problem, and finally maps the non-linear sRGB result. However, this method ignores the fact that the low brightness or overexposure problem caused by the ambient light intensity of the camera imaging occurs in the Raw-RGB stage, and the camera imaging output is usually an sRGB image after ISP non-linear processing. Directly adjusting the brightness of the sRGB image complicates the entire light adjustment processing flow.

[0004] The low brightness and overexposure problems caused by the ambient light intensity of the camera imaging occur in the Raw-RGB stage, i.e., before the ISP non-linear processing of the camera, the brightness is linearly related to the signal strength of the Raw-RGB, and is nonlinearly related to the sRGB after ISP non-linear processing. Therefore, these algorithms directly targeting sRGB require larger-scale neural network models and longer training periods to enable the network to understand the non-linear relationship. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a camera imaging optimization method and system based on a complex light environment to solve the technical problem of low brightness or overexposure when a camera faces a complex light environment, in view of the deficiencies in the prior art.

[0006] The application adopts the following technical solutions:

[0007] A camera imaging optimization method based on a complex light environment, comprising the following steps:

[0008] S1, input the nonlinear sRGB image with low brightness or overexposure into the U-Net neural network model for inference calculation to obtain a linear Raw-RGB image before camera image signal processing;

[0009] S2, input the linear Raw-RGB image obtained in step S1 into another U-Net neural network to obtain a pixel-by-pixel adjustment map for adjusting the ambient light intensity of the linear Raw-RGB image;

[0010] S3, use a neural network composed of a self-attention mechanism encoding module to infer the image signal processing parameters of the Raw-RGB image adjusted in step S2, restore the Raw-RGB image to an sRGB image with adjusted brightness through a simulated image signal processing process, and realize camera imaging optimization.

[0011] Specifically, in step S1, the U-Net convolutional neural network model is specifically:

[0012] The structure maintains the original size of the input image and infers layer by layer. During the continuous inference process, the image is gradually decomposed into its original linear Raw-RGB image form.

[0013] Specifically, in step S1, the constraint of the U-Net convolutional neural network model during training is:

[0014] The input is an sRGB image, and the output is a corresponding Raw-RGB image.

[0015] Further, the loss function Loss1 for training the U-Net convolutional neural network model is:

[0016] Loss1=L1(f(I i ),I raw )

[0017] Where L1 is a norm calculation, f is a convolutional neural network, I i is the input nonlinear sRGB image, and I raw is the linear Raw-RGB image label corresponding to the sRGB image.

[0018] Specifically, in step S2, the ambient light intensity of the linear Raw-RGB image is adjusted to obtain a Raw-RGB image D with adjusted brightness:

[0019] D=f(I i )+A

[0020] Where A is a pixel-by-pixel adjustment map item with the same size as the input image I i , and f is a U-Net convolutional neural network.

[0021] Specifically, in step S3, the image signal processing parameter inference includes white balance, color space transformation and gamma correction.

[0022] Further, the white balance and the color space transformation both use a 3*3 matrix, and the two parameters are fused into a 9-unknown-parameter matrix, and the gamma correction has a total of 10 unknown analog image signal processing parameters, and the training sample is the MIT-Adobe FiveK data set adjusted in brightness by an expert.

[0023] Still further, the 10 unknown analog image signal processing parameters are obtained through a deep learning supervised learning method.

[0024] Still further, the training loss function Loss2 is:

[0025] Loss2=L1(g(h(D),D),I t )

[0026] Wherein, L1 is a norm calculation, h is a convolutional neural network, which outputs 10 unknown analog image signal processing parameters, D is a Raw-RGB image after brightness adjustment, I t is an sRGB label image corresponding to the input image, and g is a function of performing nonlinear ISP change on the Raw-RGB image D by the 10 ISP processing parameters output by the convolutional neural network.

[0027] In a second aspect, an embodiment of the present application provides a camera imaging optimization system based on a complex lighting environment, characterized in that it comprises:

[0028] An inference module, which inputs a nonlinear sRGB image with low brightness or overexposure into a U-Net neural network model for inference calculation to obtain a linear Raw-RGB image before camera image signal processing;

[0029] An adjustment module, which inputs the linear Raw-RGB image obtained by the inference module into another U-Net neural network to obtain a pixel-by-pixel adjustment map for adjusting the linear Raw-RGB image according to the ambient light intensity;

[0030] An optimization module, which uses a neural network composed of a self-attention mechanism encoding module to infer the image signal processing parameters of the Raw-RGB image adjusted by the adjustment module, restores the Raw-RGB image to an sRGB image after brightness adjustment through an analog image signal processing process, and realizes camera imaging optimization.

[0031] Compared with the prior art, the present application has at least the following beneficial effects:

[0032] The application discloses a camera imaging optimization method based on a complex light environment, maps an sRGB image into a linear Raw-RGB image through a U-Net, processes the problem of too low or overexposure of brightness caused by environmental illumination by utilizing the linear relationship between the signal intensity of the Raw-RGB image and the brightness of the image, and has good processing effect on the adjustment of the brightness of the image.

[0033] Further, the original sRGB image and the linear Raw-RGB image have the same size, therefore, the encoder-decoder structure of the classic U-Net is used to infer the original sRGB image. In the U-Net, the original sRGB and the corresponding Raw-RGB image are used for network training, and the U-Net model after training is used to obtain the Raw-RGB image which is convenient for subsequent adjustment processing after the input of the sRGB image.

[0034] Further, the U-Net model is used to adjust the original sRGB image to obtain the linear Raw-RGB image before the camera image signal processing. In the Raw-RGB stage, the brightness of the image and the signal intensity of the Raw-RGB are linearly related, and the Raw-RGB image which has a linear relationship with the brightness of the image is directly processed, so that the whole illumination processing flow can be greatly simplified.

[0035] Further, in the training process of the U-Net, a pixel-based cross-entropy loss function is used to process each pixel point of the original sRGB, so that the pixel-level mapping relationship of the original sRGB image to the Raw-RGB image is retained to the greatest extent.

[0036] Further, after the pixel-by-pixel adjustment of the Raw-RGB image is completed, the Raw-RGB image which is not adjusted is added and fused to retain the Raw-RGB image information which may be lost in the training process, so that the subsequent image recovery processing can be better performed.

[0037] Further, after the Raw-RGB image which has completed the linear brightness adjustment is obtained, it needs to be restored into an sRGB image which can be more intuitively observed. The restoration process mainly performs white balance, color space transformation and gamma correction on the Raw-RGB image, so as to obtain the final sRGB image which has completed the brightness adjustment.

[0038] Further, in the process of white balance, color space conversion and gamma correction, a plurality of parameters are needed for calculation. White balance and color space conversion are fused and processed simultaneously, so 9 parameters are needed, and 1 parameter is used for gamma correction. These parameters cannot be directly calculated, so a deep learning supervised learning method is used, and the calculation process is embedded in the network model for calculation. We use the MIT-Adobe FiveK dataset, use the Raw-RGB image adjusted by brightness as the input of the network, and use the sRGB image corresponding to the Raw-RGB image as the label for training. The parameters for processing white balance, color space conversion and gamma correction are directly fused in the neural network model, and the error back propagation process is carried out to obtain the neural network model that can directly convert the Raw-RGB image to the sRGB image.

[0039] Further, white balance and color space conversion use 3*3 matrix, and two parameters are fused into 9 unknown parameters matrix, plus gamma correction, a total of 10 unknown analog image signal processing parameters. These parameters are difficult to calculate from the relationship between Raw-RGB and sRGB label images. Deep learning has strong adaptability, and can use the simplest and most direct way to fit the parameter calculation model that meets our requirements.

[0040] Further, in the training process of the neural network, first, the Raw-RGB image adjusted by brightness is mapped to 10 LSP processing parameters by the convolutional neural network, and the Raw-RGB image is nonlinearly transformed by using the 10 LSP parameters, and finally the sRGB image is obtained; the sRGB image obtained by inference is used to calculate the cross-entropy loss function of the pixels with the corresponding sRGB image label, and the back propagation is carried out, so as to complete the learning and training of the whole model, and obtain the neural network mapping model of the Raw-RGB image to the sRGB image.

[0041] It can be understood that the beneficial effects of the above-mentioned second aspect can be referred to the related description in the above-mentioned first aspect, which will not be repeated here.

[0042] In summary, the present application avoids directly processing the nonlinear sRGB image output by the camera, adjusts the luminance of the converted linear Raw-RGB image, and finally restores the nonlinear sRGB image, which has good ability to restore low brightness and overexposed images.

[0043] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the present application;

[0045] Figure 2 A U-Net convolutional neural network model diagram used for the method of the present application to perform sRGB image to Raw-RGB image mapping;

[0046] Figure 3 A convolutional neural network model and ambient light adjustment method flowchart for Raw-RGB inference of images used in the method of the present application;

[0047] Figure 4 An image brightness operation example for a four-circle positioning plate applied in the present application, wherein (a) is an original image obtained by a camera; (b) is a positioning plate ROI image; (c) is a positioning plate image under low brightness conditions; (d) is a positioning plate image under high exposure conditions; (e) is a positioning plate ROI image under high exposure conditions; (f) is a positioning plate ROI image adjusted by a neural network and a circle center detection result display. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0049] In the description of the present application, it should be understood that the terms "include" and "contain" indicate the existence of described features, whole, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0050] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0051] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship.

[0052] It should be understood that, although the terms first, second, third, etc. can be employed in describing the preset ranges, etc. in the embodiments of the present application, the preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range, without departing from the scope of the embodiments of the present application.

[0053] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon being determined," or "in response to determining" or "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]."

[0054] Various structural diagrams according to the disclosed embodiments of the present application are shown in the accompanying drawings. These diagrams are not drawn to scale, in which certain details are exaggerated for clarity of presentation and may omit certain details. The shapes and relative sizes of the various regions, layers, and their relative positions illustrated in the drawings are merely exemplary, and in actuality, they can deviate due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0055] The present application provides a camera imaging optimization method based on complex lighting environment. The sRGB (Standard Red Green Blue) image subjected to low brightness or overexposure due to complex lighting environment is input into a U-Net neural network for inference calculation to obtain the linear Raw-RGB image before camera image signal processing (ISP). Since the low brightness and overexposure problems caused by environmental illumination are linearly related to brightness in the Raw-RGB stage, the Raw-RGB image is input into another U-Net neural network to obtain a pixel-by-pixel adjustment map for adjusting and correcting the environmental illumination. Subsequently, the Raw-RGB image after adjustment is processed by a neural network composed of a self-attention mechanism encoding module to infer image signal processing parameters, including white balance parameters, color space transformation matrix, and gamma correction parameters. Finally, the Raw-RGB image is restored to the sRGB image with adjusted brightness through a simulated image signal processing process. The present application avoids directly processing the nonlinear sRGB image output by the camera, adjusts the illumination of the converted linear Raw-RGB image, and finally restores the nonlinear sRGB image, which has good ability to restore low brightness and overexposure images.

[0056] Please refer toFigure 1 The application discloses a camera imaging optimization method based on a complex illumination environment, which comprises the following steps:

[0057] S1, inputting a nonlinear sRGB image caused by a complex illumination environment to cause low brightness or overexposure of a camera into a U-Net neural network model for inference calculation to obtain a linear Raw-RGB image of the image before camera image signal processing (ISP) occurs;

[0058] The input image may be caused by insufficient environmental light, limited number of photons entering the camera photosensitive element, resulting in too low brightness of the camera imaging; or may be caused by too high brightness of the environmental light, reaching the physical saturation point of the camera photosensitive element, resulting in overexposure of the camera imaging; or may be caused by a partially blocked strong light source in the shooting environment, resulting in too low local brightness and local overexposure of the camera imaging.

[0059] Since the brightness deficiency or overexposure occurs in the Raw-RGB stage of camera sampling, the brightness deficiency or overexposure problem caused by the environmental illumination in the Raw-RGB space is linear, and the problem caused by the environmental illumination becomes more complex due to the nonlinear processing of the camera imaging ISP, so the nonlinear sRGB image is first restored to the corresponding linear Raw-RGB image.

[0060] Since the Raw-RGB of the camera is not easy to obtain, the linear Raw-RGB used in the application is obtained by inference of a U-Net convolutional neural network model.

[0061] Please refer to Figure 2 The U-Net convolutional neural network model for mapping the sRGB image to the Raw-RGB image is designed as:

[0062] The structure of maintaining the original size of the input image and inferring layer by layer, the image is gradually nonlinearly decomposed into the original linear Raw-RGB image form in the continuous inference process.

[0063] The U-Net convolutional neural network model is constrained as:

[0064] The input is an sRGB image, and the output is the corresponding Raw-RGB image; the corresponding training loss function is:

[0065] Loss1=L1(f(I raw ),I i ) i

[0066] Wherein, L1 is a norm calculation, f is a convolutional neural network, I raw ​The label is obtained by Invertible ISP pre-training model inference.

[0067] S2, input the inferred linear Raw-RGB image into another U-Net neural network to obtain a pixel-by-pixel adjustment map, and adjust the ambient light intensity of the linear Raw-RGB image.

[0068] The brightness adjustment process of Raw-RGB is to obtain a pixel-by-pixel adjustment map by another U-Net convolutional neural network model, and adjust Raw-RGB pixel by pixel using the map.

[0069] In the inference, an A image with the same size as the input sRGB image I i is obtained, and the final brightness-adjusted Raw-RGB image D is obtained by the following formula:

[0070] D=f(I i )+A

[0071] Where D is the Raw-RGB image after brightness adjustment, A is the pixel-by-pixel adjustment map item with the same size as the input image I i , and f is the U-Net convolutional neural network of step S1.

[0072] S3, the adjusted Raw-RGB image is used to infer the color transformation matrix and gamma correction parameters using the attention module, and the image signal processing (ISP) is performed, and finally the brightness-adjusted Raw-RGB image is restored to a non-linear sRGB image output.

[0073] Please refer to Figure 3 , the image signal processing parameter inference includes white balance, color space transformation and gamma correction, and realizes the effect of converting the Raw-RGB image to a non-linear sRGB image.

[0074] Since white balance and color space transformation use a 3x3 matrix, the two parameters are fused into a 9-unknown matrix, plus gamma correction, a total of 10 unknown analog image signal processing parameters. These parameters are obtained through deep learning supervised learning, and the training sample is the MIT-Adobe FiveK dataset adjusted by experts.

[0075] The corresponding training loss function Loss2 is:

[0076] Loss2=L1(g(h(D),D),I t )

[0077] Wherein, L1 is a norm calculation, h is a convolutional neural network, which outputs 10 unknown parameters of analog image signal processing, D is the Raw-RGB image after brightness adjustment in step S3, I t is the sRGB label image corresponding to the input image, and g is a function of nonlinear ISP change of the Raw-RGB image D by the 10 ISP processing parameters output by the convolutional neural network.

[0078] The present application avoids directly processing the nonlinear sRGB image output by the camera, adjusts the luminosity of the linear Raw-RGB image obtained by conversion, and finally restores the nonlinear sRGB image, which has good ability to recover low brightness and overexposed images.

[0079] In another embodiment of the present application, a camera imaging optimization system based on complex lighting environment is provided, which can be used to implement the camera imaging optimization method based on complex lighting environment described above. Specifically, the camera imaging optimization system based on complex lighting environment includes an inference module, an adjustment module and an optimization module.

[0080] Wherein, the inference module inputs the nonlinear sRGB image with low brightness or overexposure into the U-Net neural network model for inference calculation to obtain the linear Raw-RGB image before camera image signal processing;

[0081] The adjustment module inputs the linear Raw-RGB image obtained by the inference module into another U-Net neural network to obtain a pixel-by-pixel adjustment map, and adjusts the ambient luminosity of the linear Raw-RGB image;

[0082] The optimization module uses a neural network composed of a self-attention mechanism encoding module to infer the image signal processing parameters of the Raw-RGB image adjusted by the adjustment module, restores the Raw-RGB image to the sRGB image after adjusting the brightness through the analog image signal processing flow, and realizes camera imaging optimization.

[0083] In still another embodiment of the present application, a terminal device is provided, which comprises a processor and a memory, the memory is configured to store a computer program, the computer program comprises program instructions, and the processor is configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the camera imaging optimization method based on the complex lighting environment, which comprises:

[0084] The nonlinear sRGB image with low brightness or overexposure is input into a U-Net neural network model for inference calculation to obtain a linear Raw-RGB image before camera image signal processing; the obtained linear Raw-RGB image is input into another U-Net neural network to obtain a pixel-by-pixel adjustment map, and the linear Raw-RGB image is adjusted according to the ambient light intensity; a neural network composed of a self-attention mechanism coding module is used to infer the image signal processing parameters of the adjusted Raw-RGB image, the Raw-RGB image is restored to an sRGB image after brightness adjustment through simulation of the image signal processing flow, and camera imaging optimization is realized.

[0085] In another embodiment of the present application, the present application also provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a terminal device, used for storing programs and data. It can be understood that the computer readable storage medium herein can include a built-in storage medium in the terminal device, and of course can also include an extended storage medium supported by the terminal device. The computer readable storage medium provides a storage space, which stores an operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory.

[0086] The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the camera imaging optimization method based on the complex lighting environment in the above embodiments; the one or more instructions in the computer readable storage medium are loaded and executed by the processor as follows:

[0087] The nonlinear sRGB image with low brightness or overexposure is input into the U-Net neural network model for inference calculation to obtain a linear Raw-RGB image before camera image signal processing; the obtained linear Raw-RGB image is input into another U-Net neural network to obtain a pixel-by-pixel adjustment map for adjusting the ambient light intensity of the linear Raw-RGB image; a neural network composed of a self-attention mechanism coding module is used to infer the image signal processing parameters of the adjusted Raw-RGB image, and the Raw-RGB image is restored to an sRGB image after adjusting the brightness through the simulation of the image signal processing process, thereby realizing camera imaging optimization.

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0089] Embodiment

[0090] A camera imaging optimization method and system based on complex lighting environment aims at the target including a rectangular block commonly used in visual projects, which has four black rings of different sizes, as shown in Figure 4 (b).

[0091] The positions of the centers of the four rings in the image are detected and extracted. When the ambient light is weak, such as at night, the number of photons entering the camera photosensitive element is insufficient, resulting in a decrease in image brightness and accompanying noise, such as Figure 4 (c); when the ambient light is too strong, such as directly facing the light source, the camera photosensitive element reaches its physical saturation point, resulting in excessively high image brightness, causing overexposure, such as Figure 4 (d); therefore, in actual production, when the camera image produces low brightness or overexposure, it will interfere with the extraction and detection of the center of the algorithm, affecting production.

[0092] First, the region of interest, ROI, region of the image 4(d), that is, the image part containing the ring, is selected, and the sRGB image containing the ring, image (e), is input into the neural network model to adjust the overall brightness of the image, adjust the brightness of the image to the same image gray scale range, facilitate subsequent center detection and extraction; the adjusted image is shown in Figure 4 (f); the ring positioning plate image with adjusted brightness is used for center detection, and the result is shown in Figure 4 (f).

[0093] In summary, the camera imaging optimization method and system based on complex lighting environment uses deep learning algorithms to change and adjust the image in multiple stages, avoiding the problems of large scale, long training period, and difficulty in convergence faced by a single neural network; it has good processing effect for low brightness and overexposure under different environments, and the speed also meets the requirements.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0095] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0096] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0097] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented by other ways. For example, the above-mentioned apparatus / terminal embodiments are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0098] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0099] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0100] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer-readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0101] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks

[0102] These computer program instructions can also be stored in a computer-readable storage medium, which can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer-readable storage medium produce a manufactured product including instruction devices, which implement the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0103] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flow Figure 1 one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0104] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A method for camera imaging optimization based on complex lighting environment, characterized in that, Comprising the following steps: S1, input the nonlinear sRGB image with low brightness or overexposure into the U-Net neural network model for inference calculation to obtain the linear Raw-RGB image before camera image signal processing, and the constraint of the U-Net convolutional neural network model during training is: The execution input is an sRGB image, and the output is in the form of a corresponding Raw-RGB image, and a loss function for training a U-Net convolutional neural network model is: wherein, is a norm calculation, is a convolutional neural network, is an input non-linear sRGB image, is a linear Raw-RGB image label corresponding to the sRGB image; S2, input the linear Raw-RGB image obtained in step S1 into another U-Net neural network to obtain a pixel-by-pixel adjustment map, and adjust the ambient light intensity of the linear Raw-RGB image; S3, using a neural network composed of a self-attention mechanism coding module to process the Raw-RGB image adjusted in step S2 to infer image signal processing parameters, restore the Raw-RGB image to an sRGB image after adjusting the brightness through the simulation of the image signal processing process, realize camera imaging optimization, and the image signal processing parameter inference includes white balance, color space conversion and gamma correction, the white balance and color space conversion are all used The matrix of two parameters is fused into a 9 unknown parameter matrix, and the gamma correction has a total of 10 unknown simulation image signal processing parameters. The training sample is the MIT-Adobe FiveK dataset adjusted in brightness by experts.

2. The method for camera imaging optimization based on complex lighting environment according to claim 1, wherein, In step S1, the U-Net convolutional neural network model is specifically: The structure of maintaining the original size of the input image and inferring layer by layer, and the image is gradually decomposed into its original linear Raw-RGB image form in the continuous inference process.

3. The method of claim 1, wherein, In step S2, the ambient light intensity of the linear Raw-RGB image is adjusted to obtain a Raw-RGB image D after brightness adjustment: wherein A is a function of the input image pixel-wise adjusted map items of exactly the same size, is a U-Net convolutional neural network.

4. The method of claim 1, wherein, 10 unknown simulated image signal processing parameters are obtained by deep learning supervised learning.

5. The method for camera imaging optimization based on complex lighting environment of claim 1, wherein, Training loss function is: in, Calculate for norm 1, This is a convolutional neural network that outputs 10 unknown parameters for simulated image signal processing, where D is the brightness-adjusted Raw-RGB image. This is the sRGB label image corresponding to the input image. To apply the 10 ISP processing parameters output by the convolutional neural network to a Raw-RGB image A function that performs nonlinear ISP transformations.

6. A camera imaging optimization system based on complex lighting environment, characterized in that, Comprising: An inference module, inputting the nonlinear sRGB image with low brightness or overexposure into the U-Net neural network model for inference calculation to obtain the linear Raw-RGB image before camera image signal processing, and the constraint of the U-Net convolutional neural network model during training is: The execution input is an sRGB image, and the output is in the form of a corresponding Raw-RGB image, a loss function for training a U-Net convolutional neural network model is: wherein, is a norm computation, is a convolutional neural network, is an input non-linear sRGB image, is a linear Raw-RGB image label corresponding to the sRGB image; An adjustment module, inputting the linear Raw-RGB image obtained by the inference module into another U-Net neural network to obtain a pixel-by-pixel adjustment map, and adjusting the ambient light intensity of the linear Raw-RGB image; The optimization module uses a neural network composed of a self-attention mechanism coding module to infer the image signal processing parameters of the Raw-RGB image adjusted by the adjustment module, restores the Raw-RGB image to an sRGB image after adjusting the brightness through an analog image signal processing process, realizes camera imaging optimization, and the image signal processing parameter inference includes white balance, color space conversion and gamma correction. The matrix of the white balance and the color space conversion is used, two parameters are fused into a matrix of 9 unknown parameters, and 10 unknown analog image signal processing parameters are added to the gamma correction, and the training sample is the MIT-Adobe FiveK dataset adjusted in brightness by experts.

Images