Camera parameter estimation method, device and equipment based on deep learning

Abstract

The invention relates to the technical field of shooting, in particular to a camera parameter estimation method, device and equipment based on deep learning, and the method comprises the steps: decomposing an image in a data set into a reflectivity image and an illumination image through an improved Retinex decomposition network; generating an optical parameter feature vector based on the camera optical parameter of the image; training to obtain a camera parameter estimation model based on a convolutional neural network based on the illumination image and the optical parameter feature vector; the camera parameter estimation model is used for extracting a brightness feature and an exposure feature from the illumination image through a convolutional neural network, fusing the brightness feature and the exposure feature to generate an exposure brightness index, and matching the exposure brightness index with the optical parameter feature vector; and based on the illumination image of the target image, estimating camera optical parameters corresponding to the target image through a camera parameter estimation model. Therefore, the optical parameters of the camera can be quickly estimated based on one image, so that related personnel can perform timely adjustment.

Description

Technical Field

[0001] The present invention relates to the field of shooting technology, and particularly relates to a method, device and equipment for camera parameter estimation based on deep learning. Background Art

[0002] In current research on low-light image enhancement and exposure optimization, most technologies optimize images based on the images after camera imaging. On the one hand, there is an obvious lag, and relevant personnel cannot make appropriate adjustments to shooting; on the other hand, for the images after camera imaging, optimization can only be carried out to a certain extent, and the effect is not ideal. Therefore, there is an urgent need for a method that can provide relevant information for users to adjust shooting when the camera is shooting. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, device and equipment for camera parameter estimation based on deep learning to overcome the problem that only the images after shooting can be optimized at present.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] In the first aspect, the present application provides a method for camera parameter estimation based on deep learning, including:

[0006] Decompose the images in a preset data set into a reflectance map and an illumination map through an improved Retinex decomposition network, where the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the illumination information of the image;

[0007] Generate an optical parameter feature vector based on the camera optical parameters of the image; where the camera optical parameters include sensitivity, aperture and shutter speed;

[0008] Based on the illumination map and the optical parameter feature vector, train a camera parameter estimation model based on a convolutional neural network; where the camera parameter estimation model is used to extract brightness features and exposure features from the illumination map through a convolutional neural network, fuse the brightness features and exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector;

[0009] Estimate the camera optical parameters corresponding to the target image through the camera parameter estimation model based on the illumination map of the target image.

[0010] Further, in some embodiments of the present application, the step of decomposing the images in a preset data set into a reflectance map and an illumination map through an improved Retinex decomposition network includes:

[0011] Convert the image into latent features through an encoder;

[0012] Based on the cross-attention and self-attention mechanisms, decompose the latent features to generate the reflectance map and the illumination map.

[0013] Further, in some embodiments of the present application, generating the optical parameter feature vector based on the camera optical parameters of the image includes:

[0014] Perform non-linear equalization processing on the camera optical parameters through the non-linear equalization layer in the preset optical parameter fusion network, and perform a normalization operation on the processing result;

[0015] Input the result after the normalization process into the fully connected layer in the optical parameter fusion network to generate the optical parameter feature vector.

[0016] Further, in some embodiments of the present application, before inputting the result after the normalization process into the fully connected layer in the optical parameter fusion network, it further includes: randomly discarding some of the optical parameter feature vectors corresponding to the camera optical parameters.

[0017] Further, in some embodiments of the present application, training the camera parameter estimation model based on the convolutional neural network based on the illumination map and the optical parameter feature vector includes:

[0018] Based on the illumination map and the optical parameter feature vector, train a camera parameter estimation model based on the convolutional neural network through the mean square error loss, the consistency loss function, and the contrast loss function.

[0019] Further, in some embodiments of the present application, it further includes:

[0020] Obtain the images for training, and label the images to generate the dataset; wherein the labeling information includes the sensitivity, aperture, and shutter speed.

[0021] In a second aspect, the present application provides a camera parameter estimation device based on deep learning, including:

[0022] A decomposition module for decomposing the images in the preset dataset into a reflectance map and an illumination map through an improved Retinex decomposition network, wherein the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the illumination information of the image;

[0023] A generation module for generating an optical parameter feature vector based on the camera optical parameters of the image; wherein the camera optical parameters include the sensitivity, aperture, and shutter speed;

[0024] A training module, configured to train a camera parameter estimation model based on a convolutional neural network based on the illumination map and the optical parameter feature vector; wherein, the camera parameter estimation model is configured to extract brightness features and exposure features from the illumination map through the convolutional neural network, fuse the brightness features and the exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector;

[0025] A prediction module, configured to estimate the camera optical parameters corresponding to the target image through the camera parameter estimation model based on the illumination map of the target image.

[0026] In a third aspect, the present application provides a camera parameter estimation device based on deep learning, including a processor and a memory, the processor is connected to the memory:

[0027] Wherein, the processor is configured to call and execute a program stored in the memory;

[0028] The memory is configured to store the program, and the program is at least used to execute the above-mentioned camera parameter estimation method based on deep learning.

[0029] The present invention relates to the field of shooting technology, and specifically relates to a camera parameter estimation method, device and equipment based on deep learning. The method includes: decomposing an image in a preset data set into a reflectance map and an illumination map through an improved Retinex decomposition network, wherein the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the illumination information of the image; generating an optical parameter feature vector based on the camera optical parameters of the image; wherein, the camera optical parameters include sensitivity, aperture and shutter speed; training a camera parameter estimation model based on a convolutional neural network based on the illumination map and the optical parameter feature vector; wherein, the camera parameter estimation model is configured to extract brightness features and exposure features from the illumination map through the convolutional neural network, fuse the brightness features and the exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector; estimating the camera optical parameters corresponding to the target image through the camera parameter estimation model based on the illumination map of the target image. Description of the Drawings

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0031] Figure 1It is a schematic flowchart of a camera parameter estimation method based on deep learning provided by an embodiment of the present invention.

[0032] Figure 2 It is a schematic diagram of the principle of a camera parameter estimation method based on deep learning provided by an embodiment of the present invention.

[0033] Figure 3 It is a schematic diagram of the principle of generating an optical parameter feature vector in a camera parameter estimation method based on deep learning provided by an embodiment of the present invention.

[0034] Figure 4 It is a schematic structural diagram of a camera parameter estimation device based on deep learning provided by an embodiment of the present invention.

[0035] Figure 5 It is a schematic structural diagram of a camera parameter estimation device based on deep learning provided by an embodiment of the present invention. Detailed implementation manners

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other implementation manners obtained by those of ordinary skill in the art without making creative efforts based on the embodiments of the present invention fall within the scope protected by the present invention.

[0037] Application overview:

[0038] Currently, there is no dedicated research work on predicting the optical parameters (sensitivity (ISO), shutter speed (A), aperture (F)) of image cameras because there are the following challenges and difficulties:

[0039] (1) Joint estimation of multiple parameters is relatively complex. Since the aperture, shutter speed, and sensitivity jointly affect the image brightness, there are complex non-linear relationships between these parameters. It is relatively simple to estimate a certain one of the above parameters alone, but jointly estimating multiple parameters will significantly increase the complexity of the problem. And different camera settings may generate the same brightness image, resulting in the problem of multi-solution of parameter estimation (that is, there are different parameter combinations that can generate similar image effects).

[0040] (2) Lack of a dataset containing key photography parameters. To achieve more accurate prediction of camera optical parameters, a complete dataset containing key photography parameters such as sensitivity, shutter speed, and aperture is required. These parameters are crucial for understanding and simulating the imaging process under different lighting conditions. However, the current lack of such a dataset limits the training and generalization capabilities of deep learning models in the task of camera optical parameter estimation.

[0041] (3) In practical applications, such as guiding real-time shooting, the estimation of camera optical parameters needs to be completed quickly, while current deep learning models may be limited by computing resources.

[0042] Method embodiments:

[0043] Figure 1 is a schematic flowchart of a method for estimating camera parameters based on deep learning provided by an embodiment of the present invention. Figure 2 is a schematic diagram of the principle of a method for estimating camera parameters based on deep learning provided by an embodiment of the present invention. Please refer to Figure 1 and 2 , this embodiment may include the following steps:

[0044] S101. Decompose the images in the preset dataset into reflectance maps and illumination maps through an improved Retinex decomposition network.

[0045] Among them, the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the light information of the image.

[0046] S102. Generate an optical parameter feature vector based on the camera optical parameters of the image.

[0047] Among them, the camera optical parameters include sensitivity, aperture, and shutter speed.

[0048] In this application, the camera optical parameters include sensitivity (ISO), shutter speed (A), and aperture (F). The effects of the three on image brightness and exposure are as follows: ISO represents the sensitivity of the image sensor to light. The higher the ISO value (such as ISO1600), the higher the sensitivity, the brighter the image, but more noise may be introduced; the lower the ISO value (such as ISO 100), the lower the sensitivity, the darker the image, but the better the image quality. The shutter speed determines the exposure time of the image sensor. The slower the shutter speed (such as 1 / 30 second), the longer the exposure time, the more light enters, and the brighter the image; the faster the shutter speed (such as 1 / 1000 second), the shorter the exposure time, the less light enters, and the darker the image. The aperture controls the light entering the lens. The smaller the aperture value (such as F / 2.8), the larger the light passing aperture, the more light enters, and the brighter the image; conversely, the larger the aperture value (such as F / 16), the less light enters, and the darker the image. The interaction of the three will affect the exposure degree and brightness of the image.

[0049] In this application, first, obtain the images used to generate the estimation model and label the images to generate a dataset; among them, the annotation information includes the above camera optical parameters, namely sensitivity, aperture, and shutter speed. Then, fuse the parameters of the same image into an optical parameter feature vector.

[0050] S103. Based on the illumination map and the optical parameter feature vector, train a camera parameter estimation model based on a convolutional neural network.

[0051] Among them, the camera parameter estimation model is used to extract brightness features and exposure features from the illumination map through a convolutional neural network, fuse the brightness features and exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector.

[0052] S104. Based on the illumination map of the target image, estimate the camera optical parameters corresponding to the target image through the camera parameter estimation model.

[0053] In practical applications, relevant personnel can first simply take an image, and use the method of this application to quickly determine the camera optical parameters at the time of shooting, so as to make adjustments, providing a data basis for taking higher-quality images.

[0054] It should be noted that in practical applications, when inputting the illumination map of the target image, some camera optical parameters can also be input, and the estimation model is used to estimate the values of other unknown parameters. For example, input the illumination map of the target image and the sensitivity and shutter speed to estimate the value of the aperture, etc.

[0055] The camera parameter estimation method based on deep learning provided by the present invention decomposes the images in the preset data set into a reflectance map and an illumination map through an improved Retinex decomposition network. Among them, the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the illumination information of the image; generate an optical parameter feature vector based on the camera optical parameters of the image; among them, the camera optical parameters include sensitivity, aperture and shutter speed; based on the illumination map and the optical parameter feature vector, train a camera parameter estimation model based on a convolutional neural network; among them, the camera parameter estimation model is used to extract brightness features and exposure features from the illumination map through a convolutional neural network, fuse the brightness features and exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector; based on the illumination map of the target image, estimate the camera optical parameters corresponding to the target image through the camera parameter estimation model.

[0056] On this basis, in some embodiments of this application, decomposing the images in the preset data set into a reflectance map and an illumination map through an improved Retinex decomposition network includes: converting the image into latent features through an encoder; based on the cross-attention and self-attention mechanisms, decomposing the latent features to generate a reflectance map and an illumination map.

[0057] Specifically, when two images have exactly the same three parameters of ISO, shutter speed, and aperture, due to different scenes, it may lead to a deviation in the combination of the exposure brightness index and the brightness of the image itself during the subsequent training process, thereby affecting the accuracy of the training results. Therefore, based on the Retinex theory, this application uses the image decomposition formula, specifically I = R·L, to decompose the image, where R, L, and · represent the reflectance map, illumination map (brightness map), and element-wise multiplication, respectively.

[0058] In practical applications, the image is input into an improved Retinex decomposition network. At this time, in the improved Retinex decomposition network, the input image is converted into latent features by the encoder, and then these features are decomposed through the cross-attention and self-attention mechanisms to generate a reflectance map rich in content details (i.e., the content map) and an illumination map (i.e., the brightness map) that only contains illumination information.

[0059] It can be understood that this decomposition process is completed in the latent space, which can effectively separate the image content from the illumination conditions. Finally, a reflectance map rich in content details and an illumination map that only contains illumination information can be obtained, providing a basis for the training of subsequent estimation models.

[0060] Figure 3 is a schematic diagram of the principle of generating an optical parameter feature vector in the camera parameter estimation method based on deep learning provided by an embodiment of the present invention. As Figure 3 shown, in the embodiment of this application, generating an optical parameter feature vector based on the camera optical parameters of an image may include: performing non-linear equalization processing on the camera optical parameters through a non-linear equalization layer in a preset optical parameter fusion network, and performing a normalization operation on the processing result; then inputting the result after the normalization processing into a fully connected layer in the optical parameter fusion network to generate an optical parameter feature vector.

[0061] Specifically, the standard camera optical parameters of the images in the dataset are input into the optical parameter fusion network, and the optical parameter fusion network is used to output the optical parameter feature vectors.

[0062] In practical applications, since the magnitudes of these optical parameters vary significantly and their values generally show a non-linear distribution, in this application, the optical parameter fusion network can first process these parameters through a non-linear equalization layer. In this layer, multiple non-linear mapping functions are applied to each optical parameter for non-linear equalization processing, and then the result is normalized to the range of [0, 1] to make up for the magnitude difference between the parameters.

[0063] In addition, in order to alleviate the overfitting problem caused by the difference in the order of magnitude of parameters and insufficient training data, when the present application obtains the optical parameter feature vector using the optical parameter fusion network, it is possible to perform a random inactivation discard process on the balanced feature vector of the optical parameters, that is, randomly discard the feature vectors of some parameters with a certain probability (for example, 20%). Finally, the balanced feature vector output by the non-linear equalization layer after normalization is sent to the fully connected layer to generate the optical parameter feature vector. Figure 3 shown as z in

[0064] Furthermore, in some embodiments of the present application, based on the illumination map and the optical parameter feature vector, a camera parameter estimation model based on a convolutional neural network is trained, including: based on the illumination map and the optical parameter feature vector, through the mean square error loss, the consistency loss function, and the contrast loss function, a camera parameter estimation model based on a convolutional neural network is trained.

[0065] Specifically, as Figure 2 shown, in the present application, by using a convolutional neural network (CNN) for the obtained illumination map, two key features, namely the brightness value and the exposure value, are respectively extracted, and the above two features are fused through a fusion operation to generate a comprehensive exposure brightness index.

[0066] In addition, this index is matched with the optical parameter feature vector determined based on the camera optical parameters (aperture, shutter speed, ISO sensitivity) through the mean square error loss function, and at the same time, combined with feature separation regularization, to ensure the independence and robustness of the brightness and exposure features.

[0067] During the training process, the estimation model can optimize the feature extraction method so that images with the same camera optical parameters generate consistent exposure brightness indices.

[0068] In addition, formula (1) can be obtained through the relationship between the exposure brightness index and the three parameters.

[0069] EBN = F(ISO, A, F) (1)

[0070] According to formula (1), the various camera optical parameters of the image can be accurately calculated, and using the above estimation model, the exposure brightness index can also be estimated and the matching between the exposure brightness index and the optical parameter feature vector can be used to further determine the estimated camera optical parameters.

[0071] During the actual training process of the estimation model, multiple loss functions can be used to improve the accuracy of the estimation model. In the embodiments of the present application, by using the mean square error loss, the consistency loss function, and the contrast loss function, the accuracy of the estimation and prediction of the exposure brightness index is improved.

[0072] Specifically, the main loss function adopts the Mean Squared Error (MSE), which is used to make the predicted value of the estimation model as close as possible to the true value. The mean squared error is used to penalize the deviation between the predicted value and the true value of the estimation model, thereby guiding the estimation model to gradually optimize the prediction result. In addition, MSE is a continuous and differentiable function, which is very suitable for gradient descent optimization. The formula (2) of MSE is as follows:

[0073]

[0074] where y i is the true exposure brightness index (i.e., the true exposure brightness index calculated using the camera optical parameters of the image), is the value of the exposure brightness index predicted by the estimation model, and N is the number of samples in the current batch.

[0075] On this basis, using MSE alone may cause the estimation model to overfit on some samples, and at the same time, the performance on samples with the same or similar features may be inconsistent. That is, in the training data, there may be cases where the camera optical parameters of different images are exactly the same. Although the contents of these images are different, from the perspective of camera optical parameters, the predicted values should be the same. Therefore, this application uses a consistency loss function to constrain the consistency of the output results of the estimation model between samples with exactly the same camera optical parameters. The formula (3) of the consistency loss function is as follows:

[0076]

[0077] where 1(y i ≈y j ) means taking 1 when the parameters of samples i and j are exactly the same, otherwise 0. And and are the predictions of the exposure brightness index by the estimation model, and N is the number of sample pairs that meet the conditions. By using the consistency loss, the estimation model outputs as consistently as possible when the input parameters are exactly the same.

[0078] In addition, in the training data, there may be slight differences in the parameters of different images, such as images with ISO values of 400 and 500 respectively. Although these differences are small, from the physical law, they should correspond to a certain difference in predicted values. Therefore, this application uses a contrast loss function to constrain between samples with slightly different parameters, so that the difference in the predicted values of the estimation model is proportional to the change range of the parameters. The formula (4) of the contrast loss function is as follows:

[0079]

[0080] where 1(y i ≠y j ) is 1 when the parameters of samples i and j are different, and 0 otherwise. And is the absolute difference between the predicted values of the estimation model. By using the functional contrast loss, the estimation model can distinguish samples corresponding to different parameters and avoid too small differences between the predicted values.

[0081] In this application, the total loss function is the weighted sum of the above three loss functions, and its formula (5) is as follows:

[0082] L Total = L MSE + αL Consistency + βL Contrastive (5)

[0083] where α and β are hyperparameters that control the weights of the consistency loss and the contrast loss. Through this combination, the estimation model can not only fit the true exposure brightness value, but also improve the rationality and discrimination ability of the prediction under specific constraints.

[0084] On this basis, this application verified the performance of the method of this application, including the corresponding estimation model, by using a cloud computing platform to provide an efficient and fair experimental environment (this platform is equipped with Intel i9 3.0GHz and NVIDIA 3080 as CPU and GPU units. The operating system is Windows 11, and the deep learning platform is based on PyCharm and PyTorch).

[0085] The specific tests are as follows: Predict the ISO when the shutter speed and aperture value are known. Predict the shutter speed when the ISO and aperture value are known. Predict the aperture value when the ISO and shutter speed are known. When only the ISO is known, predict the shutter speed and aperture value at the same time. When no optical parameters are known, predict the ISO, shutter speed, and aperture value at the same time. These test scenarios cover common parameter combinations in photography and aim to verify the prediction ability and accuracy of the model under different conditions. Through these evaluations, the performance of the model in actual applications can be better understood and its performance can be further optimized. It can be seen that its performance is excellent, and the detailed results are shown in Table 1 below:

[0086] Table 1 Verification Information

[0087] optical parameter Accuracy A+F 84.6％ ISO+F 81.3％ A+ISO 82.8％ ISO 87.9％ - 75.8％

[0088] Through the model in the camera parameter estimation method provided by this application, the camera parameters when taking pictures (such as images that relevant personnel can obtain through pre-shooting) can be accurately estimated. Based on these estimated values, relevant personnel can dynamically adjust various optical parameters to optimize the exposure and brightness of the image, thereby significantly improving the image quality, especially in low-light or complex light environments. At the same time, the exposure brightness index predicted by the estimation model can be used as an important reference standard for image quality assessment to measure the exposure degree and brightness performance of the image. It can not only enable users to estimate the camera optical parameters, but also help users such as professional photographers and photography enthusiasts quickly adjust the device settings by recommending appropriate camera optical parameters, so as to take high-quality images that meet artistic and technical requirements in a short time.

[0089] Device Embodiment

[0090] Based on the same inventive concept, the present invention also provides a camera parameter estimation device based on deep learning for implementing the above method embodiment. Figure 4 It is a schematic structural diagram of the camera parameter estimation device based on deep learning provided by the embodiment of the present invention, as Figure 4 shown. The device includes:

[0091] The decomposition module 11 is used to decompose the images in the preset dataset into a reflectance map and an illumination map through an improved Retinex decomposition network, where the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the light information of the image.

[0092] The generation module 12 is used to generate an optical parameter feature vector based on the camera optical parameters of the image; where the camera optical parameters include sensitivity, aperture, and shutter speed.

[0093] The training module 13 is used to train a camera parameter estimation model based on convolutional neural network based on the illumination map and the optical parameter feature vector; where the camera parameter estimation model is used to extract brightness features and exposure features from the illumination map through convolutional neural network, fuse the brightness features and exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector.

[0094] The prediction module 14 is used to estimate the camera optical parameters corresponding to the target image through the camera parameter estimation model based on the illumination map of the target image.

[0095] Regarding the device in the above embodiment, the specific manners in which each module performs operations have been described in detail in the embodiment related to the method, and will not be elaborated here.

[0096] Device Embodiment:

[0097] Based on the same inventive concept, the present invention also provides a camera parameter estimation device based on deep learning for implementing the above method embodiments. Figure 5 FIG. is a schematic structural diagram of the camera parameter estimation device based on deep learning provided by an embodiment of the present invention. As Figure 5 shown, the camera parameter estimation device based on deep learning in this embodiment includes a processor 21 and a memory 22, and the processor 21 is connected to the memory 22. Among them, the processor 21 is used to call and execute a program stored in the memory 22; the memory 22 is used to store a program, and this program is at least used to execute the camera parameter estimation method based on deep learning in the above embodiments.

[0098] For the specific implementation scheme of the camera parameter estimation based on deep learning provided by the embodiments of the present application, reference may be made to the implementation manners of the camera parameter estimation method based on deep learning in any of the above embodiments, and details are not described herein again.

[0099] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be referred to the same or similar content in other embodiments.

[0100] It should be noted that in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "a plurality of" means at least two.

[0101] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, and this should be understood by those skilled in the technical field to which the embodiments of the present invention belong.

[0102] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following well-known technologies in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] Those of ordinary skill in the art can understand that all or part of the steps carried out in the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0104] In addition, in each of the embodiments of the present invention, the functional units can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0105] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, or the like.

[0106] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0107] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A camera parameter estimation method based on deep learning, characterized in that: include: Decomposing the images in the preset data set into a reflectance map and an illumination map through an improved Retinex decomposition network, wherein the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the illumination information of the image; Generate an optical parameter feature vector based on camera optical parameters of the image; wherein the camera optical parameters include sensitivity, aperture and shutter speed; Based on the illumination image and the optical parameter feature vector, a camera parameter estimation model based on a convolutional neural network is trained; wherein the camera parameter estimation model is used to extract brightness features and exposure features from the illumination image through a convolutional neural network, fuse the brightness features and the exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector; Based on the illumination map of the target image, the camera optical parameters corresponding to the target image are estimated by the camera parameter estimation model.

2. The camera parameter estimation method based on deep learning according to claim 1, characterized in that: The improved Retinex decomposition network is used to decompose the image in the preset data set into a reflectance map and an illumination map, including: Convert the image into latent features through an encoder; Based on the cross-attention and self-attention mechanisms, the latent features are decomposed to generate the reflectivity map and the illumination map.

3. The camera parameter estimation method based on deep learning according to claim 1, characterized in that: The step of generating an optical parameter feature vector based on the camera optical parameters of the image comprises: Performing nonlinear equalization processing on the optical parameters of the camera through a nonlinear equalization layer in a preset optical parameter fusion network, and performing a normalization operation on the processing result; The normalized result is input into the fully connected layer in the optical parameter fusion network to generate the optical parameter feature vector.

4. The camera parameter estimation method based on deep learning according to claim 3, characterized in that: Before inputting the normalized result into the fully connected layer in the optical parameter fusion network, the method further includes: randomly discarding the optical parameter feature vectors corresponding to part of the camera optical parameters.

5. The camera parameter estimation method based on deep learning according to claim 1, characterized in that: The training of a camera parameter estimation model based on a convolutional neural network based on the illumination diagram and the optical parameter feature vector comprises: Based on the illumination image and the optical parameter feature vector, a camera parameter estimation model based on a convolutional neural network is trained through a mean square error loss, a consistency loss function and a contrast loss function.

6. The camera parameter estimation method based on deep learning according to claim 1, characterized in that: Also includes: Images for training are acquired, and the images are annotated to generate the data set; wherein the annotated information includes sensitivity, aperture, and shutter speed.

7. A camera parameter estimation device based on deep learning, characterized in that: include: A decomposition module, used to decompose the images in the preset data set into a reflectance map and an illumination map through an improved Retinex decomposition network, wherein the reflectance map is used to retain the content detail information of the image, and the illumination map is used to retain the illumination information of the image; A generating module, used to generate an optical parameter feature vector based on the camera optical parameters of the image; wherein the camera optical parameters include sensitivity, aperture and shutter speed; A training module, configured to train a camera parameter estimation model based on a convolutional neural network based on the illumination diagram and the optical parameter feature vector; wherein the camera parameter estimation model is configured to extract brightness features and exposure features from the illumination diagram through a convolutional neural network, fuse the brightness features and exposure features to generate an exposure brightness index, and match the exposure brightness index with the optical parameter feature vector; The prediction module is used to estimate the camera optical parameters corresponding to the target image through the camera parameter estimation model based on the illumination map of the target image.

8. A camera parameter estimation device based on deep learning, characterized in that: The invention comprises a processor and a memory, wherein the processor is connected to the memory: Wherein, the processor is used to call and execute the program stored in the memory; The memory is used to store the program, and the program is at least used to execute the camera parameter estimation method based on deep learning according to any one of claims 1 to 6.

Images