Method, device, electronic device and storage medium for determining illumination direction

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for determining illumination direction, and the method includes: sequentially downsampling an image to be processed through multiple downsampling neural networks to obtain an image to be converted; performing channel conversion on the image to be converted through a channel conversion neural network to obtain a converted image; performing matrix transformation on the converted image to obtain a target image; determining light source pixels corresponding to the light source in the target image according to the brightness of the pixels in the target image; and calculating the illumination direction of the light source in the image to be processed according to the vector pointing from each light source pixel to the center of the target image. According to the present disclosure, the features of the image to be processed are expressed in the target image, so that the features of the illumination of the image to be processed can be accurately expressed through the target image, and then the illumination direction in the image to be processed can be directly and accurately determined according to the target. The applied model is simple and has a fast calculation speed, and is suitable for use on devices with low computing power such as mobile phones.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a method, device, electronic device and storage medium for determining a lighting direction. Background Art

[0002] In fields such as virtual reality and augmented reality, in order to present the same lighting effects as the environment on the generated virtual objects, it is necessary to first calculate the lighting direction of the light sources in the environment.

[0003] The current method of calculating the lighting direction does not calculate the lighting direction, but the spherical harmonic coefficients. The rendering process is relatively complicated compared to directly obtaining the lighting direction, and the spherical harmonic coefficients obtained are used to characterize the lighting direction, and the accuracy is not very high. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device and storage medium for determining the illumination direction, so as to at least solve the technical problems in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for determining a light direction is provided, comprising:

[0006] Downsampling the image to be processed in sequence through multiple downsampling neural networks to obtain an image to be converted, wherein the number of channels of each of the downsampling neural network output images is n times the number of channels of the downsampling neural network input image, the number of pixels in the width and height of the image to be converted is 1, and n is an integer greater than 1;

[0007] Performing channel conversion on the image to be converted by a channel conversion neural network to obtain a converted image, wherein the total number of pixels of the converted image is a target number, and the target number is the product of a preset number of channels and a preset width and a preset height;

[0008] Performing a matrix transformation on the converted image to obtain a target image, wherein the number of channels, the number of pixels in the width direction, and the number of pixels in the height direction of the target image are the preset number of channels, the preset width, and the preset height, respectively;

[0009] Determining light source pixels corresponding to the light source in the target image according to the brightness of the pixels in the target image;

[0010] The illumination direction of the light source in the image to be processed is calculated according to the vector of each light source pixel pointing to the center of the target image.

[0011] Optionally, determining light source pixels in the target image according to brightness of pixels in the target image comprises:

[0012] Determining a pixel with maximum brightness in the target image;

[0013] In the target image, pixels whose ratio of brightness to the maximum brightness is greater than a preset ratio are determined as light source pixels.

[0014] Optionally, determining light source pixels in the target image according to brightness of pixels in the target image comprises:

[0015] In the target image, pixels whose brightness is greater than or equal to a preset brightness are determined as the light source pixels.

[0016] Optionally, calculating the illumination direction of the light source in the image to be processed according to the vector pointing from each light source pixel to the center of the target image includes:

[0017] Each of the vectors is added together to obtain a sum vector, and the direction of the sum vector is used as the illumination direction.

[0018] Optionally, calculating the illumination direction of the light source in the image to be processed according to the vector pointing from each light source pixel to the center of the target image includes:

[0019] A weighted sum is performed on each of the vectors to obtain the direction of the summed vector as the illumination direction, wherein the weight of the vector is positively correlated with the brightness of the light source pixel corresponding to the vector.

[0020] Optionally, before sequentially downsampling the image to be processed through multiple downsampling neural networks, the method further includes:

[0021] When the number of pixels in the height and the number of pixels in the width of the initial image are equal, the initial image is used as the image to be processed;

[0022] In the case where the number of pixels in the height and the width of the initial image are not equal, a screenshot is obtained from the initial image as the image to be processed, wherein the number of pixels in the width and the height of the screenshot are equal, or the width and / or height of the initial image are upsampled and / or downsampled to obtain the image to be processed with the same number of pixels in the height and the width.

[0023] According to a second aspect of an embodiment of the present disclosure, a device for determining a light direction is provided, comprising:

[0024] A downsampling module is configured to perform downsampling of the image to be processed in sequence through multiple downsampling neural networks to obtain an image to be converted, wherein the number of channels of each of the downsampling neural network output images is n times the number of channels of the downsampling neural network input image, the number of pixels in the width and height of the image to be converted is 1, and n is an integer greater than 1;

[0025] A channel conversion module is configured to perform channel conversion on the image to be converted through a channel conversion neural network to obtain a converted image, wherein the total number of pixels of the converted image is a target number, and the target number is the product of a preset number of channels and a preset width and a preset height;

[0026] A matrix transformation module is configured to perform a matrix transformation on the converted image to obtain a target image, wherein the number of channels, the number of pixels in the width direction, and the number of pixels in the height direction of the target image are the preset number of channels, the preset width, and the preset height, respectively;

[0027] A pixel determination module is configured to determine light source pixels in the target image according to brightness of pixels in the target image;

[0028] The direction calculation module is configured to calculate the illumination direction of the light source in the image to be processed according to the vector of each light source pixel pointing to the center of the target image.

[0029] Optionally, the pixel determination module is configured to determine pixels with maximum brightness in the target image, and determine pixels in the target image whose ratio of brightness to the maximum brightness is greater than a preset ratio as light source pixels.

[0030] Optionally, the pixel determination module is configured to determine, in the target image, pixels having brightness greater than or equal to a preset brightness as the light source pixels.

[0031] Optionally, the direction calculation module is configured to perform, adding each of the vectors to obtain a sum vector, and taking the direction of the sum vector as the illumination direction.

[0032] Optionally, the direction calculation module is configured to perform weighted summation on each of the vectors to obtain the direction of the summed vector as the illumination direction, wherein the weight of the vector is positively correlated with the brightness of the light source pixel corresponding to the vector.

[0033] Optionally, the device further comprises:

[0034] An image determination module is configured to execute, when the number of pixels in height and width of the initial image is equal, using the initial image as the image to be processed;

[0035] And when the number of pixels in the height and width of the initial image are not equal, a screenshot is obtained from the initial image as the image to be processed, wherein the number of pixels in the width and height of the screenshot is equal, or the width and / or height of the initial image are upsampled and / or downsampled to obtain the image to be processed with an equal number of pixels in the height and width.

[0036] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including:

[0037] processor;

[0038] a memory for storing instructions executable by the processor;

[0039] The processor is configured to execute the instructions to implement the method for determining the lighting direction as described in the first aspect above.

[0040] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is proposed. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method for determining the lighting direction described in the first aspect above.

[0041] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product is configured to execute the method for determining the lighting direction described in the first aspect above.

[0042] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0043] According to the embodiments of the present disclosure, the features of the image to be processed can be gradually collected through a downsampling neural network, and then the channels of the image can be converted through a channel conversion neural network to obtain a converted image, and then the converted image can be matrix transformed, and finally the features of the image to be processed can be expressed in the target image, so that the features of the illumination of the image to be processed can be directly and accurately expressed through the target image, and then the illumination direction in the image to be processed can be accurately determined based on the target image, and the applied model is relatively simple and the calculation speed is fast, which is suitable for use on devices with low computing power such as mobile phones.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0046] Figure 1 It is a schematic diagram of a model framework in the related art.

[0047] Figure 2 It is the image captured by the rear camera in the related art.

[0048] Figure 3 It is the image captured by the front camera in the related art.

[0049] Figure 4 It is a schematic diagram of rendering results in related technologies.

[0050] Figure 5 It is a schematic flow chart of a method for determining a lighting direction according to an embodiment of the present disclosure.

[0051] Figure 6 The present invention is a schematic diagram of a model for processing an image to be processed according to a method for determining a lighting direction according to an embodiment of the present invention.

[0052] FIG. 7A to FIG. 7D The present invention is a schematic diagram of a scene for determining a lighting direction according to an embodiment of the present disclosure.

[0053] Fig. 8A and Figure 8B It is a schematic diagram of a rendering result according to an embodiment of the present disclosure.

[0054] Fig. 9 is a schematic flow chart of another method for determining a lighting direction according to an embodiment of the present disclosure.

[0055] Fig.10 is a schematic flow chart of yet another method for determining a lighting direction according to an embodiment of the present disclosure.

[0056] Fig.11 is a schematic flow chart of yet another method for determining a lighting direction according to an embodiment of the present disclosure.

[0057] Fig.12 is a schematic flow chart of yet another method for determining a lighting direction according to an embodiment of the present disclosure.

[0058] Fig.13 is a schematic flow chart of yet another method for determining a lighting direction according to an embodiment of the present disclosure.

[0059] Fig.14 It is a schematic block diagram of a lighting direction determination device according to an embodiment of the present disclosure.

[0060] Fig.15 It is a schematic block diagram of another device for determining a lighting direction according to an embodiment of the present disclosure.

[0061] Fig.16 It is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0064] The method for calculating the illumination direction in the related art can be as follows: Figure 1 The calculation is performed using the model framework shown in the figure. However, the result of the calculation is not the illumination direction, but the spherical harmonic coefficients. For example, Figure 1 The model framework shown in the figure requires collecting images through the front camera and rear camera of the mobile phone respectively, for example Figure 2 The image captured by the rear camera is shown. Figure 3 The image captured by the front camera is shown in the figure. The two images are then input into the model framework, and the image features are extracted through AlexNet. Specifically, the features are represented by a 5×5×256 matrix, which can be merged into a 5×5×512 feature vector and then converted through the Dense layer to obtain 16×3 spherical harmonic coefficients.

[0065] Since the result is the spherical harmonic coefficient, not the light direction, when rendering the object to be rendered (such as a 3D horse model), it is necessary to multiply each channel in the image where the object to be rendered is located with the spherical harmonic coefficient to obtain the rendering result of the rendered light direction. For example, the rendering result is as follows: Figure 4 As shown, the rendering process is relatively complicated to obtain the lighting direction directly, and the spherical harmonic coefficients obtained are used to characterize the lighting direction, and the accuracy is not very high.

[0066] Figure 5 This is a schematic flow chart of a method for determining the illumination direction according to an embodiment of the present disclosure. The method shown in this embodiment can be applied to terminals, such as mobile phones, tablet computers, wearable devices, etc., and can also be applied to servers, such as local servers, cloud servers, etc.

[0067] like Figure 5 As shown, the method for determining the illumination direction may include the following steps:

[0068] In step S101, the image to be processed is downsampled in sequence by multiple downsampling neural networks to obtain an image to be converted, wherein the number of channels of each of the downsampling neural network output images is n times the number of channels of the downsampling neural network input image, the number of pixels in width and height of the image to be converted is 1, and n is an integer greater than 1;

[0069] In step S102, channel conversion is performed on the image to be converted by a channel conversion neural network to obtain a converted image, wherein the total number of pixels of the converted image is a target number, and the target number is the product of a preset number of channels and a preset width and a preset height;

[0070] In step S103, a matrix transformation is performed on the matrix used to represent the converted image to obtain a target matrix used to represent the target image, wherein the number of channels, the number of pixels in the width direction, and the number of pixels in the height direction of the target image are the preset number of channels, the preset width, and the preset height, respectively;

[0071] In step S104, light source pixels corresponding to the light sources are determined in the target image according to the brightness of the pixels in the target image;

[0072] In step S105, the illumination direction of the light source in the image to be processed is calculated according to the vector pointing from each light source pixel to the center of the target image.

[0073] In one embodiment, the number of pixels in the height direction and the width of the image to be processed may be equal or unequal. In order to facilitate the design of the downsampling neural network, the image to be processed may be preprocessed so that the number of pixels in the height direction and the width are equal. Then, the preprocessed image to be processed is input into multiple downsampling neural networks. Then, the convolution step length of the downsampling neural network in the height direction and the convolution step length in the width direction may be equal, thereby ensuring that the number of pixels in the width and the height of the output image to be converted is 1.

[0074] Of course, it is also possible to directly input the images to be processed with unequal numbers of pixels in the height direction and width into multiple down-sampling neural networks. In order to make the number of pixels in the width and height of the output image to be converted equal to 1, it is necessary to set the convolution step size of the down-sampling neural network in the height direction and the convolution step size in the width direction to be unequal.

[0075] The technical solution of the present disclosure is exemplarily described below mainly under the condition that the number of pixels in the height direction and the width of the image to be processed are equal, and the number of pixels in the height direction and the width are integer multiples of n.

[0076] In one embodiment, the present disclosure is used to process the image to be processed to obtain the image to be converted, which can be as follows: Figure 6 As shown, it includes multiple downsampling neural networks, a channel conversion neural network and a matrix transformation module.

[0077] In one embodiment, the downsampling neural network and the channel conversion neural network may be convolutional neural networks or other types of neural networks. Specifically, the downsampling neural network may have a convolutional downsampling layer, which may be MaxPool2d or AveragePool2d.

[0078] In one embodiment, the downsampling neural network and the channel conversion neural network can be trained based on an open source indoor point light source dataset, and the training process can include model design, optimization, testing and evaluation. The downsampling neural network can downsample the pixels of the image on the one hand, and increase the number of channels of the image on the other hand.

[0079] For example, if the convolution step size of downsampling is n=2, then one pixel is sampled for every two pixels. Since the image consists of two dimensions, width (W) and height (H), when the image passes through a downsampling neural network, both the pixels in width and the pixels in height will be downsampled. Therefore, the resolution of the output image of the downsampling neural network is 1 / n of the resolution of the input image of the downsampling neural network. 2 , for example, when n=2, it is 1 / n 2 =1 / 4.

[0080] Since the downsampling neural network can also increase the number of channels of the image, specifically, the number of channels of the output image can be increased to n times the number of pixels in the height or width of the input image, for example, when n=2, it is 2 times.

[0081] Accordingly, the number of features contained in the input image of the down-sampling neural network is, for example, the number of channels c×the number of pixels in the width W×the number of pixels in the height H, and the number of features contained in the output image of the down-sampling neural network is n times the number of channels of the input image of the down-sampling neural network, that is, n×c. Then the number of features contained in the input image is n×c×W / n×H / n, that is, the number of pixels in height and width is reduced, but the number of channels is increased, thereby converting the features of the width dimension and the height dimension into features of the channel dimension.

[0082] Each downsampling neural network is the same, so the image to be processed can be downsampled in turn until the number of pixels in the width and height of the image to be converted is 1, and accordingly, the features of the width and height dimensions can be gradually converted into features of the channel dimension. It should be noted that the convolution step size of the downsampling neural network can be set as needed, but it should not be set too large, for example, less than or equal to 4, specifically 2, so as to avoid the problem of inaccurate conversion results caused by converting the features of the width and height dimensions into features of the channel dimension too quickly when the convolution step size is too large.

[0083] For example, if the image to be processed has 512 pixels in width and height respectively, and the number of channels is 3 (for example, red channel, green channel and blue channel respectively), then the matrix corresponding to the image to be processed can be expressed as 3×512×512, and 9 downsampling neural networks can be set to downsample the image to be processed in turn. The matrix corresponding to the input image of the first downsampling can be expressed as 6×256×256, and the matrix corresponding to the input image of the second downsampling can be expressed as 12×128×128, and so on. The final matrix corresponding to the image to be converted can be expressed as 1536×1×1, that is, there is only one pixel left in the width and height, but the number of channels becomes 1536.

[0084] On the other hand, a representation method of the target image for calculating the light source direction can be preset. For example, the target image can also be represented by the product of the number of channels and the preset width and the preset height. Then the number of channels of the target image can be set to the preset number of channels, the height to the preset width, and the width to the preset width. In order to convert the image to be converted to the representation method corresponding to the target image, the image to be converted can be first converted into a converted image. Specifically, the channel conversion neural network can be used to perform channel conversion on the image to be converted to obtain the converted image, so that the total number of pixels of the converted image is the target number, and the target number is equal to the product of the preset number of channels and the preset width and the preset height, so that the target image can be obtained based on the converted image later.

[0085] For example, the target image can be set to have 3 channels, and the number of pixels in the height direction and the number of pixels in the width direction are both 64. Then the product of 3, 64 and 64 is 12288. Therefore, the image to be converted can be converted into an image with a total number of pixels of 12288 through a channel conversion neural network, where the matrix corresponding to the converted image can be expressed as 12288×1×1, for example.

[0086] Then, the converted image can be subjected to matrix transformation. For example, the converted image can be processed by a preset matrix transformation reshape function to obtain a target matrix for representing the target image. It can be understood that the matrix 12288×1×1 is transformed into a target matrix 3×64×64 through matrix transformation.

[0087] Based on this, the gradual collection and conversion of the features of the image to be processed are completed and finally expressed in the target image, thereby realizing the expression of the illumination features of the image to be processed through the target image.

[0088] Furthermore, the light source pixels corresponding to the light source can be determined in the target image according to the brightness of the pixels in the target image. The determined light source pixels are most likely to belong to or be close to the light source. Therefore, the illumination direction of the light source in the image to be processed can be calculated based on the vector pointing to the center of the target image from each light source pixel. For example, the mean vector can be calculated by adding these vectors, and the direction of the mean vector can be used as the illumination direction of the light source in the image to be processed.

[0089] FIG. 7A to FIG. 7D The present invention is a schematic diagram of a scene for determining a lighting direction according to an embodiment of the present disclosure.

[0090] Fig. 7A and Figure 7B Both images are to be processed. The two images are located in the same environment. The actual lighting in the environment is as follows: Figure 7C As shown, the visible light source is located slightly to the upper right of center.

[0091] In one embodiment, Fig. 7A If the image to be processed is input into the downsampling neural network, the output can be as follows Fig.7D The target image shown in Figure 7B The image to be processed as shown is input into the downsampling neural network, and the output can also be as follows Fig.7D The target image shown, observe Fig.7D As can be seen from the target image shown, the light source is also located slightly to the upper right of the center.

[0092] according to Fig.7D and Figure 7C By comparison, it can be seen that according to the embodiments of the present disclosure, the obtained illumination direction is extremely close to the actual illumination direction in the environment, and therefore, the accuracy of the determined illumination direction is very high.

[0093] The embodiments shown in the present disclosure can be applied to fields such as virtual reality and augmented reality. For example, in the field of augmented reality, virtual graphics can be displayed in a real environment, and the light direction in the real environment is the same as that in the real environment. Figure 7C The same. Then according to the embodiment of the present disclosure, it can be obtained Fig.7D The lighting direction shown is the same as the lighting direction in the real environment, and the virtual image displayed in the real environment can be directly rendered according to the lighting square, such as rendering Fig. 8A and Figure 8BFrom the three-dimensional yellow duck, it can be seen that the light projected on the yellow duck is also shining from the upper right, so it accurately matches the lighting square in the real environment, achieving accurate rendering of the lighting direction.

[0094] According to the embodiments of the present disclosure, the features of the image to be processed can be gradually collected through a downsampling neural network, and then the channels of the image can be converted through a channel conversion neural network to obtain a converted image, and then the converted image can be matrix transformed, and finally the features of the image to be processed can be expressed in the target image, so as to accurately express the features of the illumination of the image to be processed through the target image, and then the illumination direction in the image to be processed can be directly and accurately determined based on the target image. For example, the difference between the calculated illumination direction and the illumination direction in the actual environment can be within 25°.

[0095] Moreover, the model used in this embodiment (the overall processing flow can be regarded as a model) is relatively simple and has a fast calculation speed. For example, the resolution of the image to be processed may not be 512×512, but may be other resolutions, such as 384×384. Then, for the image to be processed with this resolution, the amount of processed data is only 127M, which is relatively small and suitable for efficient and stable use on devices with not very strong computing power, such as mobile phones.

[0096] Fig. 9 FIG. 1 is a schematic flow chart of another method for determining a light direction according to an embodiment of the present disclosure. Fig. 9 As shown, determining the light source pixel in the target image according to the brightness of the pixel in the target image includes:

[0097] In step S1041, a pixel with maximum brightness is determined in the target image;

[0098] In step S1042, pixels in the target image whose ratio of brightness to the maximum brightness is greater than a preset ratio are determined to be light source pixels.

[0099] In one embodiment, the pixel with the maximum brightness can be first determined in the target image. For example, the values ​​of the three channels of each pixel can be added. The brightness of the pixel with the largest addition result is the maximum brightness, and the brightness of the surrounding pixels generally presents a Gaussian distribution.

[0100] Then, pixels whose ratio of brightness to maximum brightness is greater than a preset ratio can be determined in the target image, where the preset ratio can be set as needed, for example, it can be set to 1 / 2. In this case, the determined pixels are all pixels with higher brightness, which are more likely to be located near the light source, and therefore can be determined as source pixels for subsequent calculations.

[0101] Fig.10FIG. 1 is a schematic flow chart of another method for determining a light direction according to an embodiment of the present disclosure. Fig.10 As shown, determining the light source pixel in the target image according to the brightness of the pixel in the target image includes:

[0102] In step S1043, pixels in the target image whose brightness is greater than or equal to a preset brightness are determined to be the light source pixels.

[0103] In one embodiment, the pixel with the maximum brightness can be determined in the target image. For example, the values ​​of the three channels of each pixel can be added. The brightness of the pixel with the largest addition result is the maximum brightness, and the brightness of the surrounding pixels generally presents a Gaussian distribution. Then, the pixel with the maximum brightness is used as the light source pixel for subsequent calculations.

[0104] Fig.11 FIG. 1 is a schematic flow chart of another method for determining a light direction according to an embodiment of the present disclosure. Fig.11 As shown, the calculation of the illumination direction of the light source in the image to be processed according to the vector of each light source pixel pointing to the center of the target image includes:

[0105] In step S1051, each of the vectors is added together to obtain a sum vector, and the direction of the sum vector is used as the illumination direction.

[0106] In one embodiment, for each vector pointing from a light source pixel to the center of the target image, the vectors may be added together to calculate a sum vector, and the direction of the sum vector may be used as the illumination direction.

[0107] For example, if 9 light source pixels are determined, and the vectors pointing to the center of the target image are (1,1), (1,2), (1,3), (2,1), (2,2), (2,3), (3,1), (3,2), and (3,3), then the sum vector obtained by adding these 9 vectors is (18,18), and the direction of the sum vector can be simplified to (1,1).

[0108] Since the direction of the sum vector obtained by adding each of the vectors is used as the illumination direction, the calculation process only involves the sum operation of the vectors, and the calculation process is relatively simple, which is conducive to ensuring a faster processing speed.

[0109] Fig.12 FIG. 1 is a schematic flow chart of another method for determining a light direction according to an embodiment of the present disclosure. Fig.12 As shown, the calculation of the illumination direction of the light source in the image to be processed according to the vector of each light source pixel pointing to the center of the target image includes:

[0110] In step S1052, a weighted sum is performed on each of the vectors to obtain the direction of the summed vector as the illumination direction, wherein the weight of the vector is positively correlated with the brightness of the light source pixel corresponding to the vector.

[0111] In one embodiment, for each vector pointing to the center of the target image from a light source pixel, a weighted summation can be performed to obtain a summation vector, wherein the weight of the vector in the weighted summation process can be set to be positively correlated with the brightness of the light source pixel corresponding to the vector. Accordingly, the direction of the summation vector can be inclined to the direction of the pixel with larger brightness, which is more consistent with the opposite direction of lighting in the actual environment.

[0112] For example, two light source pixels A and B are determined, the vector pointing from pixel A to the center of the target image is (1,1), and the vector pointing from pixel B to the center of the target image is (1,2). Pixel A has a higher brightness, so a larger de-weighted value, such as 2 / 3, can be set for pixel A, and a smaller weight, such as 1 / 3, can be set for pixel B. Then the sum vector after weighted summation of vector (1,1) and vector (1,2) is (1,4 / 3).

[0113] Since the weights used for weighting in the weighted summation process can be flexibly set as needed, it is convenient to adjust the method of calculating the summation vector by adjusting the weights.

[0114] Fig.13 FIG. 1 is a schematic flow chart of another method for determining a light direction according to an embodiment of the present disclosure. Fig.13 As shown, before sequentially downsampling the image to be processed through multiple downsampling neural networks, the method further includes:

[0115] In step S106, when the number of pixels in the height and the width of the initial image are equal, the initial image is used as the image to be processed;

[0116] In step S107, when the number of pixels in the height and the width of the initial image are not equal, a screenshot is obtained from the initial image as the image to be processed, wherein the number of pixels in the width and the height of the screenshot are equal, or the width and / or height of the initial image are upsampled and / or downsampled to obtain the image to be processed with the same number of pixels in the height and the width.

[0117] In one embodiment, in order to simplify the downsampling neural network, the convolution step size of the downsampling neural network in the height direction and the width direction can be set to be the same. On this basis, in order to make the number of pixels in the width and height of the image to be converted obtained after multiple downsampling neural networks be 1, it is necessary to ensure that the number of pixels in the height and width of the image to be processed is equal.

[0118] Therefore, for the initial image to be input into multiple downsampling neural networks, it can be determined whether the width and height of the image are equal.

[0119] When the number of pixels in the height and the number of pixels in the width of the initial image are equal, the initial image is used as the image to be processed, and step S101 is executed to sequentially downsample the image to be processed through multiple downsampling neural networks.

[0120] When the number of pixels in the height and width of the initial image are not equal, a screenshot can be obtained from the image to be processed, wherein the number of pixels in the width and height of the screenshot are equal. In this case, the screenshot can be used as the image to be processed, and then step S101 is executed to downsample the image to be processed in turn through multiple downsampling neural networks. Accordingly, it can be ensured that the image to be converted obtained after being processed by multiple downsampling neural networks has a width and height of 1, which is convenient for subsequent operations such as channel conversion and matrix transformation on the image.

[0121] In addition, when the number of pixels in the height and the number of pixels in the width of the initial image are not equal, the width and / or height of the initial image may be upsampled and / or downsampled.

[0122] For example, if the number of pixels on the height of the initial image is three times the number of pixels on the width, then the pixels on the height of the initial image can be downsampled once according to the convolution step size of 3, and then the sampled pixels are used as the pixels on the height of the image to be processed, and the pixels on the width of the initial image are used as the pixels on the width of the image to be processed, thereby obtaining the image to be processed with the same number of pixels on the height and width.

[0123] Then the pixels on the width of the initial image can be upsampled once according to the convolution step size 3, and then the sampled pixels are used as the pixels on the width of the image to be processed, and the pixels on the height of the initial image are used as the pixels on the height of the image to be processed, so as to obtain the image to be processed with the same number of pixels in height and width.

[0124] Based on this, it can be ensured that the width and height of the image to be converted after being processed by multiple downsampling neural networks are both 1, which is convenient for subsequent operations such as channel conversion and matrix transformation on the image.

[0125] Corresponding to the above-mentioned embodiment of the method for determining the illumination direction, the present disclosure also proposes an embodiment of the device for determining the illumination direction.

[0126] Fig.14This is a schematic block diagram of a device for determining the direction of illumination according to an embodiment of the present disclosure. The device shown in this embodiment can be applied to terminals, such as mobile phones, tablet computers, wearable devices, etc., and can also be applied to servers, such as local servers, cloud servers, etc.

[0127] like Fig.14 As shown, the illumination direction determining device may include:

[0128] The downsampling module 101 is configured to perform downsampling of the image to be processed in sequence through multiple downsampling neural networks to obtain an image to be converted, wherein the number of channels of each of the downsampling neural network output images is n times the number of channels of the downsampling neural network input image, the number of pixels in the width and height of the image to be converted is 1, and n is an integer greater than 1;

[0129] The channel conversion module 102 is configured to perform channel conversion on the image to be converted through a channel conversion neural network to obtain a converted image, wherein the total number of pixels of the converted image is a target number, and the target number is the product of a preset number of channels and a preset width and a preset height;

[0130] The matrix transformation module 103 is configured to perform a matrix transformation on the converted image to obtain a target image, wherein the number of channels, the number of pixels in the width direction, and the number of pixels in the height direction of the target image are the preset number of channels, the preset width, and the preset height, respectively;

[0131] The pixel determination module 104 is configured to determine light source pixels in the target image according to the brightness of the pixels in the target image;

[0132] The direction calculation module 105 is configured to calculate the illumination direction of the light source in the image to be processed according to the vector of each light source pixel pointing to the center of the target image.

[0133] Optionally, the pixel determination module is configured to determine pixels with maximum brightness in the target image, and determine pixels in the target image whose ratio of brightness to the maximum brightness is greater than a preset ratio as light source pixels.

[0134] Optionally, the pixel determination module is configured to determine, in the target image, pixels having brightness greater than or equal to a preset brightness as the light source pixels.

[0135] Optionally, the direction calculation module is configured to perform, adding each of the vectors to obtain a sum vector, and taking the direction of the sum vector as the illumination direction.

[0136] Optionally, the direction calculation module is configured to perform weighted summation on each of the vectors to obtain the direction of the summed vector as the illumination direction, wherein the weight of the vector is positively correlated with the brightness of the light source pixel corresponding to the vector.

[0137] Fig.15 FIG. 1 is a schematic block diagram of another device for determining a light direction according to an embodiment of the present disclosure. Fig.15 As shown, the device also includes:

[0138] The image determination module 106 is configured to execute, when the number of pixels in the height and the number of pixels in the width of the initial image are equal, using the initial image as the image to be processed;

[0139] And when the number of pixels in the height and width of the initial image are not equal, a screenshot is obtained from the initial image as the image to be processed, wherein the number of pixels in the width and height of the screenshot is equal, or the width and / or height of the initial image are upsampled and / or downsampled to obtain the image to be processed with an equal number of pixels in the height and width.

[0140] The present disclosure also provides an electronic device, including:

[0141] processor;

[0142] a memory for storing instructions executable by the processor;

[0143] The processor is configured to execute the instructions to implement the method for determining the lighting direction as described in any of the above embodiments.

[0144] An embodiment of the present disclosure further provides a storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method for determining the illumination direction described in any of the above embodiments.

[0145] An embodiment of the present disclosure further provides a computer program product, which is configured to execute the method for determining the lighting direction described in any of the above embodiments.

[0146] Fig.16 1 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. For example, the electronic device 1600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0147] Reference Fig.16, the electronic device 1600 may include one or more of the following components: a processing component 1602 , a memory 1604 , a power component 1606 , a multimedia component 1608 , an audio component 1610 , an input / output (I / O) interface 1612 , a sensor component 1614 , and a communication component 1616 .

[0148] The processing component 1602 generally controls the overall operation of the electronic device 1600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1602 may include one or more processors 1620 to execute instructions to complete all or part of the steps of the above-mentioned XX method. In addition, the processing component 1602 may include one or more modules to facilitate the interaction between the processing component 1602 and other components. For example, the processing component 1602 may include a multimedia module to facilitate the interaction between the multimedia component 1608 and the processing component 1602.

[0149] The memory 1604 is configured to store various types of data to support operations on the electronic device 1600. Examples of such data include instructions for any application or method operating on the electronic device 1600, contact data, phone book data, messages, pictures, videos, etc. The memory 1604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0150] The power supply component 1606 provides power to the various components of the electronic device 1600. The power supply component 1606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1600.

[0151] The multimedia component 1608 includes a screen that provides an output interface between the electronic device 1600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1608 includes a front camera and / or a rear camera. When the electronic device 1600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0152] The audio component 1610 is configured to output and / or input audio signals. For example, the audio component 1610 includes a microphone (MIC), and when the electronic device 1600 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1604 or sent via the communication component 1616. In some embodiments, the audio component 1610 also includes a speaker for outputting audio signals.

[0153] I / O interface 1612 provides an interface between processing component 1602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0154] The sensor assembly 1614 includes one or more sensors for providing various aspects of status assessment for the electronic device 1600. For example, the sensor assembly 1614 can detect the open / closed state of the electronic device 1600, the relative positioning of components, such as the display and keypad of the electronic device 1600, and the sensor assembly 1614 can also detect the position change of the electronic device 1600 or a component of the electronic device 1600, the presence or absence of user contact with the electronic device 1600, the orientation or acceleration / deceleration of the electronic device 1600, and the temperature change of the electronic device 1600. The sensor assembly 1614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 1614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0155] The communication component 1616 is configured to facilitate wired or wireless communication between the electronic device 1600 and other devices. The electronic device 1600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 1616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0156] In one embodiment of the present disclosure, the electronic device 1600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned method for determining the direction of illumination.

[0157] In an embodiment of the present disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1604 including instructions, and the instructions can be executed by a processor 1620 of an electronic device 1600 to complete the above-mentioned method for determining the illumination direction. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0159] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0160] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0161] The method and device provided in the embodiments of the present disclosure are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A method for determining a light direction, characterized in that: include: Downsampling the image to be processed in sequence through multiple downsampling neural networks to obtain an image to be converted, wherein the number of channels of each of the downsampling neural network output images is n times the number of channels of the downsampling neural network input image, the number of pixels in the width and height of the image to be converted is 1, and n is an integer greater than 1; Performing channel addition on the image to be converted by a channel conversion neural network to obtain a converted image, wherein the total number of pixels of the converted image is a target number, and the target number is the product of a preset number of channels and a preset width and a preset height; Performing a matrix transformation on the converted image to obtain a target image, wherein the number of channels, the number of pixels in the width direction, and the number of pixels in the height direction of the target image are the preset number of channels, the preset width, and the preset height, respectively; Determining light source pixels corresponding to the light source in the target image according to the brightness of the pixels in the target image; According to the vector pointing from each light source pixel to the center of the target image, the illumination direction of the light source in the image to be processed is calculated; the calculation of the illumination direction of the light source in the image to be processed according to the vector pointing from each light source pixel to the center of the target image includes: adding each of the vectors to obtain a sum vector, and taking the direction of the sum vector as the illumination direction; or performing weighted summation on each of the vectors to obtain the direction of the sum vector as the illumination direction, wherein the weight of the vector is positively correlated with the brightness of the light source pixel corresponding to the vector.

2. The method according to claim 1, characterized in that Determining light source pixels in the target image according to the brightness of pixels in the target image comprises: Determining a pixel with maximum brightness in the target image; In the target image, pixels whose ratio of brightness to the maximum brightness is greater than a preset ratio are determined as light source pixels.

3. The method according to claim 1, characterized in that Determining light source pixels in the target image according to the brightness of pixels in the target image comprises: In the target image, pixels whose brightness is greater than or equal to a preset brightness are determined as the light source pixels.

4. The method according to any one of claims 1 to 3, characterized in that Before sequentially downsampling the image to be processed through multiple downsampling neural networks, the method further includes: When the number of pixels in the height and the number of pixels in the width of the initial image are equal, the initial image is used as the image to be processed; In the case where the number of pixels in the height and the width of the initial image are not equal, a screenshot is obtained from the initial image as the image to be processed, wherein the number of pixels in the width and the height of the screenshot are equal, or the width and / or height of the initial image are upsampled and / or downsampled to obtain the image to be processed with the same number of pixels in the height and the width.

5. A device for determining a light direction, characterized in that: include: A downsampling module is configured to perform downsampling of the image to be processed in sequence through multiple downsampling neural networks to obtain an image to be converted, wherein the number of channels of each of the downsampling neural network output images is n times the number of channels of the downsampling neural network input image, the number of pixels in the width and height of the image to be converted is 1, and n is an integer greater than 1; A channel conversion module is configured to perform channel addition on the image to be converted through a channel conversion neural network to obtain a converted image, wherein the total number of pixels of the converted image is a target number, and the target number is the product of a preset number of channels and a preset width and a preset height; A matrix transformation module is configured to perform a matrix transformation on the converted image to obtain a target image, wherein the number of channels, the number of pixels in the width direction, and the number of pixels in the height direction of the target image are the preset number of channels, the preset width, and the preset height, respectively; A pixel determination module is configured to determine light source pixels in the target image according to brightness of pixels in the target image; The direction calculation module is configured to calculate the illumination direction of the light source in the image to be processed according to the vector of each light source pixel pointing to the center of the target image; the direction calculation module is configured to add each of the vectors to obtain a sum vector, and use the direction of the sum vector as the illumination direction; or perform weighted summation on each of the vectors to obtain the direction of the sum vector as the illumination direction, wherein the weight of the vector is positively correlated with the brightness of the light source pixel corresponding to the vector.

6. The device according to claim 5, characterized in that The pixel determination module is configured to determine pixels with maximum brightness in the target image, and determine pixels in the target image whose ratio of brightness to the maximum brightness is greater than a preset ratio as light source pixels.

7. The device according to claim 5, characterized in that The pixel determination module is configured to determine, in the target image, pixels whose brightness is greater than or equal to a preset brightness as the light source pixels.

8. The device according to any one of claims 5 to 7, characterized in that Also includes: An image determination module is configured to execute, when the number of pixels in height and width of the initial image is equal, using the initial image as the image to be processed; And when the number of pixels in the height and width of the initial image are not equal, a screenshot is obtained from the initial image as the image to be processed, wherein the number of pixels in the width and height of the screenshot is equal, or the width and / or height of the initial image are upsampled and / or downsampled to obtain the image to be processed with an equal number of pixels in the height and width.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for determining the lighting direction as claimed in any one of claims 1 to 4.

10. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for determining the illumination direction as claimed in any one of claims 1 to 4.

Images