Lighting simulation network model and lighting simulation image processing method

Abstract

The present invention discloses a lighting simulation network model and a lighting simulation image processing method, wherein a model image preprocessing module is used to perform preliminary feature extraction on a model input image to obtain a three-channel feature map and a single-channel feature map; a low-light enhancement module is used to perform feature extraction on the three-channel feature map and the single-channel feature map based on a lighting attention block to obtain a low-light enhanced image, and the low-light enhanced image is converted into a multi-dimensional vector through a conversion function; a channel conversion module is used to perform channel conversion operations on the model input image to obtain a single-channel conversion feature map; a splicing module is used to perform interpolation operations on the single-channel conversion feature map and the multi-dimensional vector to obtain a multi-channel interpolation image, and the multi-channel interpolation image is spliced ​​with the model input image to obtain a lighting simulation enhanced image. The present invention realizes the processing of high-definition images with as little computation as possible, and realizes lighting simulation of images to improve image quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a lighting simulation network model and a lighting simulation image processing method. Background Art

[0002] At present, the underlying visual field based on artificial intelligence technology is developing rapidly. Many open source technologies represented by low-light image enhancement have achieved good results in academia. However, people's requirements for image resolution are also increasing in today's era. Real-time update of high-definition images such as 4K is also a requirement that people are eager to achieve. However, high-definition images occupy a large amount of memory, and deployment on mobile terminals is a major problem currently faced. At the same time, there is still no image processing method specifically designed to simulate professional lighting. If low-light image enhancement technology is used, it can only improve the brightness of the image, and cannot handle detailed areas of the image such as reflections and shadows.

[0003] Therefore, there is an urgent need for a lighting simulation model and an image processing method that can be implemented on a mobile terminal and can simultaneously perform lighting simulation on high-definition images. Summary of the invention

[0004] The purpose of this application is to provide a lighting simulation network model and a lighting simulation image processing method, which are used to solve the problem that there are currently no lighting simulation models and image processing methods for lighting simulation of high-definition images, and that the current high-definition image processing occupies a large amount of content and cannot be deployed on mobile terminals.

[0005] In a first aspect, the present application provides a lighting simulation network model, comprising an image preprocessing module, a low-light enhancement module and a splicing module connected in sequence, and a channel conversion module connected to the splicing module;

[0006] The image preprocessing module is used to perform preliminary feature extraction on the model input image to obtain a three-channel feature map and a single-channel feature map;

[0007] The low-light enhancement module is used to extract features from the three-channel feature map and the single-channel feature map based on the light attention block to obtain a low-light enhanced image, and convert the low-light enhanced image into a multi-dimensional vector through a conversion function;

[0008] The channel conversion module is used to perform a channel conversion operation on the model input image to obtain a single channel conversion feature map;

[0009] The splicing module is used to perform an interpolation operation on the single-channel conversion feature map and the multi-dimensional vector to obtain a multi-channel interpolation image, and to splice the multi-channel interpolation image with the model input image to obtain a light simulation enhanced image.

[0010] In one embodiment of the present application, the image preprocessing module includes a feature map acquisition unit and a single-channel downsampling unit and a three-channel downsampling unit respectively connected to the feature map acquisition unit;

[0011] The single-channel downsampling unit is used to downsample the model input image and convert it into a single channel to obtain a single-channel sampling image;

[0012] The three-channel downsampling unit is used to downsample the model input image to obtain a three-channel sampling image;

[0013] The feature map acquisition unit is used to perform channel superposition on the single-channel sampling map and the three-channel sampling map according to the channel dimension to obtain a spliced ​​image, input the spliced ​​images into two convolutional layers respectively to obtain the single-channel feature map and the three-channel initial feature map respectively, and realize the pixel-level dot multiplication between the three-channel sampling map and the three-channel initial feature map through a jump connection to obtain the three-channel feature map.

[0014] In one embodiment of the present application, the low-light enhancement module is an IGT module with a U-shaped architecture, and the IGT module includes an input convolution layer, a downsampling unit and an upsampling unit. The downsampling unit and the upsampling unit both include multiple convolution layers and multiple light attention blocks.

[0015] In one embodiment of the present application, the lighting simulation network model also includes a brightness extraction module, which is used to perform single-channel brightness feature extraction on the model input image to obtain a single-channel brightness image, and output the single-channel brightness image to the output end of the upsampling unit to achieve the addition of the output image of the upsampling unit output end and the single-channel brightness image.

[0016] In one embodiment of the present application, the illumination attention block is an IGAB unit; or the illumination attention block is an ITAB unit, and the illumination attention block is an ITAB unit, each of which includes a normalization layer and a multi-head attention block;

[0017] The input feature map of the current ITAB unit is normalized and combined with the single channel feature map. Figure 1 The input is input to the multi-head attention block to obtain a preliminary output feature image, and the preliminary output feature image is spliced ​​with the input feature map through a jump connection to obtain an output feature map of the current ITAB unit;

[0018] Among them, the preliminary output feature image acquisition process includes: the multi-head attention block converts the input feature map after the normalization layer into input feature map data through a deformation operation, and performs a QK matrix operation on the input feature map data to obtain a QK matrix operation result. The multi-head attention block also converts the single-channel feature map into single-channel feature map data through a deformation operation, performs a V matrix operation on the single-channel feature map data and the input feature map data and passes through a layer of convolution operation to obtain a convolution result, multiplies the QK matrix operation result and the convolution result pixel by pixel to obtain a pixel multiplication result, and passes the pixel multiplication result through a fully connected layer and a position encoding layer in sequence to obtain a preliminary output feature map.

[0019] In one embodiment of the present application, the feature extraction processes of the image preprocessing module and the low-light enhancement module are both operated on a 256-resolution dimension.

[0020] In one embodiment of the present application, the channel conversion module uses a convolution kernel operator with a convolution kernel size of 1 to perform a channel conversion operation on the model input image to obtain the single channel conversion feature map.

[0021] In a second aspect, the present application also provides a light simulation image processing method, comprising:

[0022] Acquire image data to be processed, and input the image data to be processed into a trained lighting simulation network model to output lighting simulation image data;

[0023] The training process of the trained lighting simulation network model is as follows:

[0024] A training image data set is obtained, and the light simulation network model is trained based on the training image data set to obtain a trained light simulation network model.

[0025] In one embodiment of the present application, the light simulation image processing method is applied to a mobile terminal.

[0026] In one embodiment of the present application, the image preprocessing module and the low-light enhancement module in the trained lighting simulation network model are set to form a feature extraction module in the trained lighting simulation network model structure, and the feature extraction module, the channel conversion module and the splicing module in the trained lighting simulation network model are set to run in three memory spaces, two memory spaces or one memory space in the mobile terminal.

[0027] Compared with the prior art, one or more embodiments of the above scheme may have the following advantages or beneficial effects:

[0028] The lighting simulation network model provided by the embodiment of the present invention adopts an image preprocessing module to realize preliminary feature extraction of the model input image, and performs pixel-level dot multiplication between the three-channel sampling image and the three-channel initial feature map through jump connection to improve the image brightness of the three-channel feature map; the low-light enhancement module introduces the self-attention operation in the VisionTransformer to realize the lighting enhancement of the image while reducing the calculation amount of the model as much as possible and improving the operation rate of the model; and the feature extraction process is operated on the 256 resolution dimension, which also greatly reduces the calculation amount of high-resolution images; the above-mentioned channel conversion module uses a convolution kernel operator with a convolution kernel of 1 to perform channel conversion on the model input image, further reducing the model calculation amount; finally, a multi-dimensional vector is used to interpolate the single-channel conversion feature map, so that the final lighting simulation enhanced image achieves an effect very close to the label map. The model of the present invention realizes the processing of high-definition images with as little calculation amount as possible, and realizes lighting simulation of images to improve image quality. The light simulation image processing method provided by the embodiment of the present invention can be applied to mobile terminals, and the use scenario can be light simulation image processing in the implementation process of the mobile terminal, such as high-quality light simulation scene presentation during live broadcasting, etc. In order to reduce the computing power of each memory space in the mobile terminal, the feature extraction module, channel conversion module and splicing module in the above method can be respectively set in three content spaces, two content spaces or one memory space in the mobile terminal to run, further improving the feasibility of applying the light simulation network model to the mobile terminal.

[0029] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 Shown is a structural schematic diagram of the lighting simulation network model described in an embodiment of the present application.

[0032] Figure 2 Shown is a specific structural schematic diagram of the lighting simulation network model described in an embodiment of the present application.

[0033] Figure 3 Shown is a second specific structural schematic diagram of the lighting simulation network model described in an embodiment of the present application.

[0034] Figure 4 Shown is a third specific structural schematic diagram of the lighting simulation network model described in the embodiment of the present application.

[0035] Figure 5 Display as Figure 3 Schematic diagram of the structure of an illumination attention block in .

[0036] Figure 6 Display as Figure 5 Schematic diagram of the structure of a multi-head attention block in the medium illumination attention block.

[0037] Figure 7 Shown is a schematic diagram of the process of applying the trained light simulation network model to a mobile terminal in the light simulation image processing method described in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will describe the implementation methods of the present invention in detail with reference to the accompanying drawings and embodiments, so that the implementation process of how the present invention applies technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly. It should be noted that as long as there is no conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other, and the technical solutions formed are all within the protection scope of the present invention.

[0039] The following embodiments of the present application provide a lighting simulation network model and a lighting simulation image processing method, which are used to solve the problem that there are currently no lighting simulation models and image processing methods for lighting simulation of high-definition images, and that the current high-definition image processing occupies a large amount of content and cannot be deployed on mobile terminals.

[0040] The following will describe in detail the principles and implementation methods of a lighting simulation network model and a lighting simulation image processing method of this embodiment in conjunction with the accompanying drawings, so that those skilled in the art can understand the lighting simulation network model and the lighting simulation image processing method of this embodiment without creative work.

[0041] like Figure 1 As shown, this embodiment provides a lighting simulation network model, which includes an image preprocessing module, a low-light enhancement module, a stitching module and a channel conversion module, wherein the image preprocessing module, the low-light enhancement module and the stitching module are connected in sequence, and the channel conversion module is connected to the stitching module.

[0042] The image preprocessing module is used to perform preliminary feature extraction on the model input image to obtain a three-channel feature map and a single-channel feature map. The low-light enhancement module is used to extract features from the three-channel feature map and the single-channel feature map based on the illumination attention block to obtain a low-light enhanced image, and convert the low-light enhanced image into a multi-dimensional vector through a conversion function. The channel conversion module is used to perform channel conversion operations on the model input image to obtain a single-channel conversion feature map. The splicing module is used to interpolate the single-channel conversion feature map with the multi-dimensional vector to obtain a multi-channel interpolated image, and splice the multi-channel interpolated image with the model input image to obtain a light simulation enhanced image.

[0043] The lighting simulation network model of the embodiment of the present invention uses an image preprocessing module to extract preliminary features of the model input image; the low-light enhancement module realizes the lighting enhancement of the image while reducing the model's calculation amount as much as possible, thereby improving the model's operation rate; the channel conversion module uses a convolution kernel operator to perform channel conversion on the model input image, further reducing the model's operation amount; and finally, a multi-dimensional vector is used to perform an interpolation operation on the single-channel conversion feature map, so that the final lighting simulation enhanced image achieves an effect very close to the label map. The model of the present invention realizes the processing of high-definition images with the least amount of calculation possible, and realizes lighting simulation of images to improve image quality.

[0044] refer to Figures 2 to 4 As shown, in one embodiment, the image preprocessing module specifically includes a single-channel downsampling unit, a three-channel downsampling unit and a feature map acquisition unit. The single-channel downsampling unit and the three-channel downsampling unit are respectively connected to the feature map acquisition unit. The specific single-channel downsampling unit is mainly used to perform single-channel downsampling on the model input image, that is, downsampling the model input image and converting it into a single-channel image to obtain a single-channel sampling map. The three-channel downsampling unit is mainly used to downsample the model input image to obtain a three-channel sampling map. The feature map acquisition unit is used to splice the single-channel sampling map and the three-channel sampling map together according to the channel dimension in a cat manner to obtain a spliced ​​image, and then input the spliced ​​image into two convolutional layers respectively to obtain a single-channel feature map lightupFeature and a three-channel initial feature map respectively. At the same time, the feature map acquisition unit also sets a jump connection between the three-channel sampling map and the three-channel initial feature map to supplement the three-channel initial feature map with pixels based on the three-channel sampling map to obtain the final three-channel feature map lightupMap. A jump connection is set between the three-channel sampling map and the three-channel initial feature map to realize the pixel-level dot product between the two, so as to achieve the purpose of improving the brightness of the three-channel feature map image.

[0045] refer to Figure 2 and Figure 3As shown, in one embodiment, the low-light enhancement module is an IGT module with a U-shaped architecture, and the IGT module includes an input convolution layer, a downsampling unit, and an upsampling unit. The input convolution layer is used to further extract features from the three-channel feature map to obtain a feature map lit-up. The downsampling unit and the upsampling unit both include multiple convolution layers and multiple illumination attention blocks. The specific downsampling unit includes a first convolution layer, an illumination attention block, a second convolution layer, two illumination attention blocks, a third convolution layer, and two illumination attention blocks in sequence. Furthermore, the first convolution layer can be set to a 3×3 convolution with a step size of 2, the second convolution layer can be set to a 3×3 convolution with a step size of 2, and the third convolution layer can be set to a 3×3 convolution with a step size of 2. The upsampling unit and the downsampling unit are symmetrical structures, and the specific upsampling unit includes a fourth convolution layer, two illumination attention blocks, a fifth convolution layer, two illumination attention blocks, and a sixth convolution layer in sequence. Furthermore, the fourth convolution layer can be set to a 3×3 convolution with a stride of 2, and the fifth convolution layer can be set to a 3×3 convolution with a stride of 2. It should be noted that the stride and convolution kernel size of the above convolution layers can also be set to other reasonable values ​​based on actual conditions, and no fixed restrictions are imposed on them here.

[0046] refer to Figure 4 As shown, in one embodiment, the low-light enhancement module is an IGT module with a U-shaped architecture, and the IGT module includes an input convolution layer, a downsampling unit, and an upsampling unit. The input convolution layer is used to further extract features from the three-channel feature map to obtain a feature map lit-up. Both the downsampling unit and the upsampling unit include multiple convolution layers and multiple illumination attention blocks. The specific downsampling unit includes a first convolution layer, an illumination attention block, a second convolution layer, and two illumination attention blocks in sequence. Furthermore, the first convolution layer can be set to a 3×3 convolution with a step size of 2, and the second convolution layer can be set to a 3×3 convolution with a step size of 2. The upsampling unit and the downsampling unit are symmetrical structures, and the specific upsampling unit includes a third convolution layer, two illumination attention blocks, and a fourth convolution layer in sequence. Furthermore, the third convolution layer can be set to a 3×3 convolution with a step size of 2, and the fourth convolution layer can be set to a 3×3 convolution with a step size of 2. A convolution layer is set between the model input image and the output of the upsampling unit. The convolution layer processes the model input image into a single-channel brightness image, specifically extracts brightness features, and the single-channel brightness image only displays brightness information. After the single-channel brightness image is added to the output image of the upsampling unit output, the final light simulation enhanced image has better performance in both brightness and shadow areas.

[0047] Figure 4 Structure compared to Figure 2 and Figure 3The structure further reduces the amount of calculation of the lighting simulation network model, so that the trained lighting simulation network model can achieve real-time application with high refresh rate and high resolution when running on a mobile terminal.

[0048] The low-light enhancement module adds the self-attention operation in VisionTransformer, continuously extracts features during the downsampling process, and uses cat-style jump connections to increase feature information in the decoder (i.e., upsampling unit) during the downsampling process. The low-light enhancement module also sets a long jump connection between the feature map lit-up and the output of the upsampling unit to output a low-light enhanced image enhanceimage. The low-light enhancement module then converts the low-light enhanced image enhanceimage into a multidimensional vector of B, 12, 8, 16, 16 dimensions through the conversion function view, where B represents the size of the batchsize.

[0049] It is worth noting that the feature extraction operations in the above-mentioned image preprocessing module and low-light enhancement module are all operated on the 256-resolution dimension, which can greatly reduce the computational complexity of image processing for high-definition images such as 2k resolution or 4k resolution input images.

[0050] In one embodiment, the illumination attention block in the low-light enhancement module is an IGAB unit. Since the IGAB unit is an existing unit, the specific structure thereof is not further described in this embodiment.

[0051] refer to Figure 3-Figure 6 As shown, in one embodiment, the illumination attention block in the low-light enhancement module can also be set as an ITAB unit. Specifically, the ITAB unit includes a normalization layer and a multi-head attention block. Compared with the IGAB unit, the ITAB unit has the characteristics of fewer parameters and lighter weight; it further facilitates the deployment of the trained lighting simulation network model on the mobile terminal, can realize the rapid processing of high-definition images on the mobile terminal, and improves the data processing speed of the trained lighting simulation network model.

[0052] refer to Figure 5 As shown, further, the data transmission process in the illumination attention block is as follows: the input feature map Iinput of the current ITAB unit is input into the multi-head attention block TA-block together with the single-channel feature map lightupFeature after passing through the normalization layer LN to obtain a preliminary output feature image, and finally the preliminary output feature image is spliced ​​with the input feature map through a jump connection to obtain the output feature map Ioutput of the current ITAB unit.

[0053] refer to Figure 6As shown, the process of obtaining the preliminary output feature image includes: the multi-head attention block TA-block converts the input feature map Oinput after the normalization layer LN into input feature map data through a reshape deformation operation, and performs a QK matrix operation on the input feature map data to obtain the QK matrix operation result; at the same time, the multi-head attention block TA-block also converts the single-channel feature map lightupFeature into single-channel feature map data through a reshape deformation operation, and then performs a V matrix operation on the single-channel feature map data and the input feature map data and passes through a layer of convolution operation to obtain the convolution result, and multiplies the QK matrix operation result and the convolution result pixel by pixel to obtain the pixel multiplication result, and finally passes the pixel multiplication result through the fully connected layer fc and the position encoding layer P in sequence to obtain the preliminary output feature map Ooutput.

[0054] refer to Figures 2 to 4 As shown, in one embodiment, the channel conversion module is used to perform pixel operation on the model input image, specifically using a convolution operator with a convolution kernel of 1 to perform channel conversion operation to obtain a single channel conversion feature map. Using a convolution operator with a convolution kernel of 1 has the effect of saving computing power and further reducing the amount of model calculation.

[0055] refer to Figures 2 to 4 As shown, in one embodiment, the splicing module performs a slicing interpolation operation on the single-channel conversion feature map and the multi-dimensional vector to obtain a multi-channel interpolation image, and then applies a splicing operation on the multi-channel interpolation image and the model input image to obtain the final light simulation enhanced image. The slicing operation is an interpolation operation in three dimensions. Such an interpolation operation will interpolate the image into multiple channels. The specific number of channels selected needs to correspond to the subsequent apply splicing operation. The apply splicing operation is a matrix multiplication, and its purpose is to multiply the color information rgain channel, ggain channel and bgain channel with the model input image in the form of a matrix to obtain the final light simulation enhanced image.

[0056] Since there are no operations such as convolution that require back propagation in the slicing interpolation operation and the apply splicing operation in the splicing module, the splicing module can be independent of the part of the network that needs to be learned in the lighting simulation network model, and can be deployed as a post-processing module in the memory space of the CPU, NPU, GPU, etc. in the mobile terminal. It should also be noted that the part of the network that needs to be learned in the lighting simulation network model can also be separated into two parts and set up independently in the mobile terminal. Specifically, the image preprocessing module and the low-light enhancement module can be used as independent parts, and the channel conversion module can be used as an independent part.

[0057] Furthermore, since the lighting simulation network model of the embodiment of the present invention achieves more efficient feature extraction, that is, feature extraction is performed at a low resolution and remapped back to a high-resolution image to achieve efficient operation of the model, the calculation amount of the model is reduced as much as possible during the processing of high-definition images, and the operation rate of the model is improved. Therefore, the lighting simulation network model of the embodiment of the present invention overcomes the problem that the memory occupied by the current high-definition image processing is too large to be deployed on a mobile terminal, and the processed image cannot be applied in real time due to the huge amount of calculation in the image processing process. The lighting simulation network model of the embodiment of the present invention can be implemented to run in a server that can complete training. Further, based on the learning characteristics of the lighting simulation network model, when the lighting simulation network model runs in the server, in order to reduce the computing power of each memory space in the server, the feature extraction total module, channel conversion module and splicing module formed by the image preprocessing module and the low-light enhancement module in the lighting simulation network model can be respectively set in three memory spaces in the server for operation, or the above modules can be set in two memory spaces in the server according to the memory size for operation, or the above modules can be set in one memory space in the server for operation, so as to provide better support for achieving real-time operation.

[0058] The lighting simulation network model of the present invention uses an image preprocessing module to realize preliminary feature extraction of the model input image, and performs pixel-level dot multiplication between the three-channel sampling image and the three-channel initial feature map through jump connection to improve the image brightness of the three-channel feature map; the low-light enhancement module introduces the self-attention operation in the VisionTransformer to realize the lighting enhancement of the image while reducing the calculation amount of the model as much as possible, thereby improving the operation rate of the model; and the feature extraction process is operated on the 256 resolution dimension, which also greatly reduces the calculation amount of high-resolution images; the above-mentioned channel conversion module uses a convolution kernel operator with a convolution kernel of 1 to perform channel conversion on the model input image, further reducing the model calculation amount; finally, a multi-dimensional vector is used to interpolate the single-channel conversion feature map, so that the final lighting simulation enhanced image achieves an effect very close to the label map. The model of the present invention realizes the processing of high-definition images with as little calculation amount as possible, and realizes lighting simulation of images to improve image quality.

[0059] This embodiment also provides a light simulation image processing method, which specifically includes:

[0060] The image data to be processed is obtained, and the image data to be processed is input into the trained lighting simulation network model to output the lighting simulation image data.

[0061] The training process of the further trained lighting simulation network model is as follows:

[0062] A training image data set is obtained, and the lighting simulation network model described in the above embodiment is trained based on the training image data set to obtain a trained lighting simulation network model.

[0063] The process of acquiring the training image data set is as follows: collecting the data set, which can take the live broadcast scene as the main application scenario, that is, the collected data comes from the real live broadcast scene; then the collected data is cleaned and selected and other preliminary processing is performed; the data after preliminary processing is sent to the data enhancement module, and the data enhancement module can perform one or more processing operations such as normalization, contrast and brightness adjustment, translation conversion or random rotation on the data, and then obtain the enhanced image data; the enhanced image data can provide better generalization, and finally the enhanced image data is divided into training and testing to form a training image data set.

[0064] It should be noted that the collected data can also be applied in other suitable scenarios. The live broadcast scenario is only one application scenario in which the light simulation image processing method of the embodiment of the present invention can be implemented. The light simulation image processing method of the present invention does not impose fixed restrictions on its application scenario.

[0065] In one embodiment, the light simulation image processing method can be applied to a mobile terminal, that is, the trained light simulation network model is deployed in the mobile terminal. Figure 7 As shown in the figure, the specific deployment process of deploying the trained lighting simulation network model on the mobile terminal is as follows: after training the lighting simulation network model based on the training image data set, the effect can be verified by test data. When the verification effect meets the effect requirements, the trained lighting simulation network model will be converted into an onnx file, and then the model in the onnx file will be quantized using the post-training quantization strategy; the quantized onnx will also be verified by test data. When the quantized effect can meet the scene usage, the onnx file will be further converted into an om file, and then the om file will be uploaded to the development board. After the development code is compiled, the om model will be tested. The om model will return a binary bin file, and the bin file will be converted into png for verification. If it does not meet the requirements, it will return to onnx to gradually test the accuracy loss. If the png meets the effect and the speed meets the actual use, the om model will be sent to the embedded development to develop the actual project. The om file can be an AI model file supported by Huawei HiSilicon development board or an AI model file supported by other brands of development boards.

[0066] Furthermore, since the trained light simulation network model reduces the amount of calculation of the model and improves the operation rate of the model, based on the learning characteristics of the light simulation network model, the feature extraction module, channel conversion module and splicing module formed by the image preprocessing module and the low-light enhancement module in the trained light simulation network model can be respectively set in three memory spaces in the mobile terminal for operation, or they can be set in two memory spaces in the mobile terminal for operation according to the memory size, or they can be set in one memory space in the mobile terminal for operation, which provides better support for realizing real-time operation.

[0067] Taking the 1080P image processing process as an example, the light simulation image processing method implemented in the mobile terminal according to the embodiment of the present invention can realize the real-time operation of the 1080P image at 30 / 60 frames per second on the mobile terminal.

[0068] The light simulation image processing method provided by the embodiment of the present invention can be applied to mobile terminals, and the use scenario can be light simulation image processing in the implementation process of the mobile terminal, such as high-quality light simulation scene presentation during live broadcasting, etc. In order to reduce the computing power of each memory space in the mobile terminal, the feature extraction module, channel conversion module and splicing module in the above method can be respectively set in three content spaces, two content spaces or one memory space in the mobile terminal to run, further improving the possibility of applying the light simulation network model to the mobile terminal.

[0069] Of course, the embodiments disclosed in the present invention are as above, but the contents described are only embodiments adopted for facilitating the understanding of the present invention, and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the protection scope of the present invention shall still be subject to the scope defined by the attached claims.

Claims

1. A lighting simulation network model, characterized in that: It includes an image preprocessing module, a low-light enhancement module and a splicing module connected in sequence, and a channel conversion module connected to the splicing module; The image preprocessing module is used to perform preliminary feature extraction on the model input image to obtain a three-channel feature map and a single-channel feature map; The low-light enhancement module is used to extract features from the three-channel feature map and the single-channel feature map based on the light attention block to obtain a low-light enhanced image, and convert the low-light enhanced image into a multi-dimensional vector through a conversion function; The channel conversion module is used to perform a channel conversion operation on the model input image to obtain a single channel conversion feature map; The splicing module is used to perform an interpolation operation on the single-channel conversion feature map and the multi-dimensional vector to obtain a multi-channel interpolation image, and splice the multi-channel interpolation image with the model input image to obtain a light simulation enhanced image; Among them, the interpolation operation is a slicing interpolation operation; the channel conversion module uses a convolution kernel operator with a convolution kernel size of 1 to perform a channel conversion operation on the model input image to obtain the single channel conversion feature map.

2. The model according to claim 1, characterized in that The image preprocessing module includes a feature map acquisition unit and a single-channel downsampling unit and a three-channel downsampling unit respectively connected to the feature map acquisition unit; The single-channel downsampling unit is used to downsample the model input image and convert it into a single channel to obtain a single-channel sampling image; The three-channel downsampling unit is used to downsample the model input image to obtain a three-channel sampling image; The feature map acquisition unit is used to perform channel superposition on the single-channel sampling map and the three-channel sampling map according to the channel dimension to obtain a spliced ​​image, input the spliced ​​image into two convolutional layers respectively to obtain the single-channel feature map and the three-channel initial feature map respectively, and realize the pixel-level dot multiplication between the three-channel sampling map and the three-channel initial feature map through a jump connection to obtain the three-channel feature map.

3. The model according to claim 1, characterized in that The low-light enhancement module is an IGT module with a U-shaped architecture. The IGT module includes an input convolution layer, a downsampling unit and an upsampling unit. The downsampling unit and the upsampling unit each include multiple convolution layers and multiple light attention blocks.

4. The model according to claim 3, characterized in that The lighting simulation network model also includes a brightness extraction module, which is used to perform single-channel brightness feature extraction on the model input image to obtain a single-channel brightness image, and output the single-channel brightness image to the output end of the upsampling unit to achieve the addition of the output image of the upsampling unit output end and the single-channel brightness image.

5. The model according to claim 3, characterized in that The illumination attention block is an IGAB unit; or the illumination attention block is an ITAB unit, each of which includes a normalization layer and a multi-head attention block; The input feature map of the current ITAB unit is input into the multi-head attention block together with the single-channel feature map after passing through the normalization layer to obtain a preliminary output feature image, and the preliminary output feature image is spliced ​​with the input feature map through a jump connection to obtain the output feature map of the current ITAB unit; Among them, the preliminary output feature image acquisition process includes: the multi-head attention block converts the input feature map after the normalization layer into input feature map data through a deformation operation, and performs a QK matrix operation on the input feature map data to obtain a QK matrix operation result. The multi-head attention block also converts the single-channel feature map into single-channel feature map data through a deformation operation, performs a V matrix operation on the single-channel feature map data and the input feature map data and passes through a layer of convolution operation to obtain a convolution result, multiplies the QK matrix operation result and the convolution result pixel by pixel to obtain a pixel multiplication result, and passes the pixel multiplication result through a fully connected layer and a position encoding layer in sequence to obtain a preliminary output feature map.

6. The model according to claim 1, characterized in that The feature extraction processes of the image preprocessing module and the low-light enhancement module both operate on a 256-resolution dimension.

7. A lighting simulation image processing method, comprising: Acquire image data to be processed, and input the image data to be processed into a trained lighting simulation network model to output lighting simulation image data; The training process of the trained lighting simulation network model is as follows: A training image data set is obtained, and the lighting simulation network model according to any one of claims 1 to 6 is trained based on the training image data set to obtain a trained lighting simulation network model.

8. The processing method according to claim 7, characterized in that: The light simulation image processing method is applied to a mobile terminal.

9. The processing method according to claim 8, characterized in that: The image preprocessing module and the low-light enhancement module in the trained lighting simulation network model are set to form a feature extraction general module in the trained lighting simulation network model structure. The feature extraction general module, the channel conversion module and the splicing module in the trained lighting simulation network model are set to run in three memory spaces, two memory spaces or one memory space in the mobile terminal.

Images