An image enhancement method and related device

Abstract

The present application provides an image enhancement method and related devices, which can accurately display the image information to be highlighted in the enhanced target image, thereby improving the image enhancement effect. The method of the present application includes: obtaining a first feature of the texture information of the target image and a second feature of the color information of the target image through an image processing model; performing a first fusion process on the first feature and the second feature through the image processing model to obtain a third feature of the texture information, and performing a second fusion process on the first feature and the second feature through the image processing model to obtain a fourth feature of the color information; generating a first bilateral grid according to the third feature, and generating a second bilateral grid according to the fourth feature; performing enhancement processing on the target image according to the first bilateral grid and the second bilateral grid to obtain the enhanced target image.

Description

Technical Field

[0001] This application relates to the technical field of artificial intelligence (AI), and in particular, to an image enhancement method and related devices thereof. Background Art

[0002] With the rapid development of computer technology, people use a large number of various images in daily life and production activities. To improve the image quality, image enhancement can be performed. Image enhancement refers to a processing method that highlights certain information of an image according to specific requirements and weakens or removes some unnecessary information.

[0003] Currently, after obtaining a target image to be enhanced, the features of the target image can be extracted through an image processing model, a corresponding bilateral grid can be generated based on the features, and then the target image can be enhanced using the bilateral grid, so as to obtain the enhanced target image.

[0004] However, only generating a bilateral grid based on the features of the entire target image, the considered factors are relatively single, that is, the information about the target image contained in the bilateral grid is relatively one-sided. Therefore, the enhanced target image obtained based on the bilateral grid cannot accurately display some image information that needs to be highlighted, resulting in poor image enhancement effect. Summary of the Invention

[0005] Embodiments of this application provide an image enhancement method and related devices thereof, which can enable the enhanced target image to accurately display the image information that needs to be highlighted, thereby improving the image enhancement effect.

[0006] The first aspect of the embodiments of this application provides an image enhancement method, characterized in that the method includes:

[0007] After obtaining a target image to be enhanced, the target image can be preprocessed first to obtain the texture information of the target image and the color information of the target image.

[0008] After obtaining the texture information of the target image and the color information of the target image, this part of the information can be input into an image processing model to obtain the first feature of the texture information of the target image and the second feature of the color information of the target image through the image processing model.

[0009] After obtaining the first feature of the texture information and the second feature of the color information, the image processing model can perform a first fusion process on the first feature and the second feature, that is, incorporate the second feature on the basis of the first feature, so as to obtain a third feature. The third feature can be used to characterize the texture information and can also be used to characterize the correlation between the texture information and the color information. Correspondingly, the image processing model can also perform a second fusion process on the first feature and the second feature, that is, incorporate the first feature on the basis of the second feature, to obtain a fourth feature. The fourth feature can be used to characterize the color information and can also be used to characterize the correlation between the texture information and the color information.

[0010] After obtaining the third feature of the texture information and the fourth feature of the color information, a first bilateral grid of the texture information can be generated according to the third feature, and a second bilateral grid of the color information can be generated according to the fourth feature.

[0011] After obtaining the first bilateral grid of the texture information and the second bilateral grid of the color information, the target image can be enhanced according to the first bilateral grid and the second bilateral network to obtain an enhanced target image.

[0012] It can be seen from the above method that since the third feature is obtained by incorporating the second feature of the color information on the basis of the first feature of the texture information, and the fourth feature is obtained by incorporating the first feature of the texture information on the basis of the second feature of the color information, the first bilateral grid generated based on the third feature not only contains the texture information of the target image, but also contains the correlation between the texture information and the color information. The second bilateral grid generated based on the fourth feature not only contains the color information of the target image, but also contains the correlation between the texture information and the color information. The information about the target image contained in the two bilateral grids is relatively comprehensive. Then, the enhanced target image obtained based on the first bilateral grid and the second bilateral grid can accurately display the image information to be highlighted (such as texture, color, saturation, brightness, dynamic range, contrast, etc.), thereby improving the image enhancement effect.

[0013] In a possible implementation, the image processing model includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. Obtaining the first feature of the texture information of the target image and the second feature of the color information of the target image through the image processing model includes: performing convolutional processing on the texture information of the target image through N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; performing convolutional processing on the color information of the target image through N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers. In the foregoing implementation, through the N convolutional layers, multi-level convolutional processing can be respectively performed on the texture information and color information of the target image to correspondingly obtain N first sub-features and N second sub-features. Therefore, subsequent feature fusion processing can be implemented based on the N first sub-features and N second sub-features.

[0014] In a possible implementation, the image processing model further includes N first fusion layers and N second fusion layers. The first first fusion layer is connected to the first convolutional layer, the i-th first fusion layer is connected to the i-th second fusion layer, and the i-th convolutional layer is connected to the i-th second fusion layer, where i = 1,..., N and N is an integer greater than or equal to 1. The first feature and the second feature are subjected to first fusion processing by the image processing model to obtain a third feature of texture information, and the first feature and the second feature are subjected to second fusion processing by the image processing model to obtain a fourth feature of color information, including: performing convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; performing normalization processing on the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; performing convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1,..., N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature. In the foregoing implementation, the outputs of the N convolutional layers are respectively used as the inputs of the N second fusion layers, the outputs of the N second fusion layers are respectively used as the inputs of the N first fusion layers, and the output of the (i - 1)-th first fusion layer is used as the input of the i-th first fusion layer. In this way, with the mutual cooperation of the N convolutional layers, the N first fusion layers, and the N second fusion layers, the N first sub-features and the N second sub-features can be fused and interacted multiple times in the order from low level to high level. Therefore, the finally obtained third feature can not only be used to represent texture information, but also be used to represent the correlation between texture information and color information. Similarly, the fourth feature can not only be used to represent color information, but also be used to represent the correlation between texture information and color information.

[0015] In a possible implementation, before obtaining the first feature of the texture information of the target image and the second feature of the color information of the target image through the image processing model, the method further includes: after obtaining the target image, first performing downsampling processing on the target image to obtain a downsampled target image, where the resolution of the downsampled target image is less than the resolution of the original target image. Then, performing separation processing on the downsampled target image to obtain the texture information of the target image and the color information of the target image. For example, the downsampled target image can be subjected to a certain degree of Gaussian blur processing, and then the high-frequency information therein can be removed to obtain the color information of the target image, and the texture information of the target image can be obtained by subtracting the color information from the downsampled target image, and so on.

[0016] In a possible implementation, the image processing model further includes an image reconstruction layer. Enhancing the target image according to the first bilateral grid and the second bilateral network to obtain the enhanced target image includes: First, performing grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and performing grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid. The grid reconstruction processing is used to remove the less influential information in the first bilateral grid and the second bilateral grid and retain the more influential information. Next, performing upsampling processing on the third bilateral grid to obtain a fifth bilateral grid, and performing upsampling processing on the fourth bilateral grid to obtain a sixth bilateral grid. Wherein, the resolution of the fifth bilateral grid is the same as the resolution of the original target image, and the resolution of the sixth bilateral grid is also the same as the resolution of the original target image. Then, performing an affine transformation on the target image according to the fifth bilateral grid to obtain a fifth feature, and performing an affine transformation on the target image according to the sixth bilateral grid to obtain a sixth feature. Wherein, the resolution of the fifth feature is the same as the resolution of the original target image, and the resolution of the sixth feature is also the same as the resolution of the original target image. Finally, performing image reconstruction processing on the fifth feature and the sixth feature through the image reconstruction layer to obtain the enhanced target image.

[0017] In a possible implementation, grid reconstruction processing is performed on the first bilateral grid to obtain a third bilateral grid, and grid reconstruction processing is performed on the second bilateral grid to obtain a fourth bilateral grid, including: performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral grid respectively to obtain the first bilateral grid after height processing; performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral grid respectively to obtain the first bilateral grid after width processing; performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing; performing weighted summation processing on the first bilateral grid after height processing, the first bilateral grid after width processing, and the first bilateral grid after depth processing to obtain a third bilateral grid; performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral grid respectively to obtain the second bilateral grid after height processing; performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral grid respectively to obtain the second bilateral grid after width processing; performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid respectively to obtain the second bilateral grid after depth processing; performing weighted summation processing on the second bilateral grid after height processing, the second bilateral grid after width processing, and the second bilateral grid after depth processing to obtain a fourth bilateral grid. In the foregoing implementation, Tucker decomposition and Tucker reconstruction can be respectively performed on the first bilateral grid and the second bilateral grid, which is beneficial to filtering out some less useful information in these two bilateral grids. Then, in the process of subsequent processing of the target image based on the obtained third bilateral grid and fourth bilateral grid, problems such as noise and incorrect boundaries generated by the bilateral grid in the target image can be avoided.

[0018] In a possible implementation, performing a mesh reconstruction process on the first bilateral mesh to obtain a third bilateral mesh, and performing a mesh reconstruction process on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral mesh to obtain the first bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral mesh to obtain the first bilateral mesh after width processing; after obtaining the first bilateral mesh after height processing and the first bilateral mesh after width processing, weighted summation processing can be performed on these two bilateral meshes to obtain the third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral mesh to obtain the second bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral mesh to obtain the second bilateral mesh after width processing; performing weighted summation processing on the second bilateral mesh after height processing and the second bilateral mesh after width processing to obtain the fourth bilateral mesh.

[0019] In a possible implementation, performing a mesh reconstruction process on the first bilateral mesh to obtain a third bilateral mesh, and performing a mesh reconstruction process on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral mesh to obtain the first bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral mesh to obtain the first bilateral mesh after depth processing; after obtaining the first bilateral mesh after height processing and the first bilateral mesh after depth processing, weighted summation processing can be performed on these two bilateral meshes to obtain the third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral mesh to obtain the second bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral mesh to obtain the second bilateral mesh after depth processing; performing weighted summation processing on the second bilateral mesh after height processing and the second bilateral mesh after depth processing to obtain the fourth bilateral mesh.

[0020] In a possible implementation, grid reconstruction processing is performed on the first bilateral grid to obtain a third bilateral grid, and grid reconstruction processing is performed on the second bilateral grid to obtain a fourth bilateral grid, including: performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral grid respectively to obtain the first bilateral grid after width processing; performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing; after obtaining the first bilateral grid after width processing and the first bilateral grid after depth processing, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral grid respectively to obtain the second bilateral grid after width processing; performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid respectively to obtain the second bilateral grid after depth processing; performing weighted summation processing on the second bilateral grid after width processing and the second bilateral grid after depth processing to obtain a fourth bilateral grid.

[0021] In a possible implementation, grid reconstruction processing is performed on the first bilateral grid to obtain a third bilateral grid, and grid reconstruction processing is performed on the second bilateral grid to obtain a fourth bilateral grid, including: performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral grid respectively to obtain a third bilateral grid; performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0022] In a possible implementation, grid reconstruction processing is performed on the first bilateral grid to obtain a third bilateral grid, and grid reconstruction processing is performed on the second bilateral grid to obtain a fourth bilateral grid, including: performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral grid respectively to obtain a third bilateral grid; performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0023] In a possible implementation, grid reconstruction processing is performed on the first bilateral grid to obtain a third bilateral grid, and grid reconstruction processing is performed on the second bilateral grid to obtain a fourth bilateral grid, including: performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid respectively to obtain a third bilateral grid; performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0024] In a possible implementation, the image processing model further includes a feature extraction layer that upsamples the third bilateral grid to obtain a fifth bilateral grid and upsamples the fourth bilateral grid to obtain a sixth bilateral grid, which includes: after obtaining the target image, the target image can be input into the feature extraction layer of the image processing model to perform feature extraction processing on the target image through the second feature extraction layer to obtain a texture guidance map and a color guidance map. Then, interpolation processing can be performed on the third bilateral grid with reference to the texture guidance map to obtain the fifth bilateral grid, and interpolation processing can be performed on the fourth bilateral grid with reference to the color guidance map to obtain the sixth bilateral grid.

[0025] In a possible implementation, image reconstruction processing is performed on the fifth feature and the sixth feature through an image reconstruction layer to obtain an enhanced target image, which includes: first, third fusion processing is performed on the fifth feature and the sixth feature through the image reconstruction layer to obtain a seventh feature. Then, convolution processing is performed on the seventh feature through the image reconstruction layer to obtain the enhanced target image. For example, the image reconstruction layer includes a normalization layer and a sequentially connected convolution layer. After receiving the fifth feature and the sixth feature, the normalization layer performs normalization processing on the fifth feature and the sixth feature and sends the obtained seventh feature to the convolution layer. After receiving the seventh feature from the normalization layer, the convolution layer can perform convolution processing on the seventh feature to obtain the enhanced target image.

[0026] In a possible implementation, the third fusion processing includes at least one of normalization processing, summation processing, multiplication processing, concatenation processing, and concatenation convolution processing.

[0027] The second aspect of the embodiments of the present application provides a model training method, which includes: obtaining an image to be processed and a real image after image enhancement of the image to be processed; obtaining a first feature of the texture information of the image to be processed and a second feature of the color information of the image to be processed through the model to be trained; performing first fusion processing on the first feature and the second feature through the model to be trained to obtain a third feature of the texture information, and performing second fusion processing on the first feature and the second feature through the model to be trained to obtain a fourth feature of the color information; generating a first bilateral grid according to the third feature and generating a second bilateral grid according to the fourth feature; performing enhancement processing on the image to be processed according to the first bilateral grid and the second bilateral network to obtain a predicted image; obtaining a target loss according to the predicted image, the real image, and the processed predicted image, where the target loss is used to indicate the difference between the predicted image and the real image, and the difference between the predicted image and the processed predicted image; updating the model parameters of the model to be trained according to the target loss until the model training condition is satisfied to obtain the image processing model.

[0028] The image processing model obtained based on the above method has the ability to enhance the image to be processed. During the image enhancement process, the image processing model can incorporate the second feature of color information based on the first feature of texture information to obtain a third feature, and the image processing model can incorporate the first feature of texture information based on the second feature of color information to obtain a fourth feature. Therefore, the first bilateral grid generated based on the third feature not only contains the texture information of the image to be processed, but also contains the correlation between the texture information and the color information. The second bilateral grid generated based on the fourth feature not only contains the color information of the image to be processed, but also contains the correlation between the texture information and the color information. The information about the image to be processed contained in the two bilateral grids is relatively comprehensive. Then, the enhanced image to be processed obtained based on the first bilateral grid and the second bilateral grid can accurately display the image information that needs to be highlighted, thereby improving the image enhancement effect.

[0029] In a possible implementation, the processed predicted image is the predicted image after pixel flipping processing and blurring processing.

[0030] In a possible implementation, the model to be trained includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. Obtaining the first feature of the texture information of the image to be processed and the second feature of the color information of the image to be processed through the model to be trained includes: performing convolutional processing on the texture information of the image to be processed through N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; performing convolutional processing on the color information of the image to be processed through N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

[0031] In a possible implementation, the model to be trained further includes N first fusion layers and N second fusion layers. Performing a first fusion process on the first feature and the second feature through the model to be trained to obtain the third feature of the texture information, and performing a second fusion process on the first feature and the second feature through the model to be trained to obtain the fourth feature of the color information includes: performing convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; performing normalization processing on the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; performing convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1,..., N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

[0032] In a possible implementation, before obtaining the first feature of the texture information of the image to be processed and the second feature of the color information of the image to be processed through the model to be trained, the method further includes: obtaining the image to be processed; performing downsampling processing on the image to be processed to obtain the downsampled image to be processed; and performing separation processing on the downsampled image to be processed to obtain the texture information of the image to be processed and the color information of the image to be processed.

[0033] In a possible implementation, the model to be trained further includes an image reconstruction layer. Enhancing the image to be processed according to the first bilateral grid and the second bilateral network to obtain a predicted image includes: performing grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and performing grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid, where the grid reconstruction processing is used to remove some information in the bilateral grid; performing upsampling processing on the third bilateral grid to obtain a fifth bilateral grid, and performing upsampling processing on the fourth bilateral grid to obtain a sixth bilateral grid; performing affine transformation on the image to be processed according to the fifth bilateral grid to obtain a fifth feature, and performing affine transformation on the image to be processed according to the sixth bilateral grid to obtain a sixth feature; and performing image reconstruction processing on the fifth feature and the sixth feature through the image reconstruction layer to obtain a predicted image.

[0034] In a possible implementation, performing grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and performing grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral grid to obtain the first bilateral grid after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral grid to obtain the first bilateral grid after width processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid to obtain the first bilateral grid after depth processing; performing weighted summation processing on the first bilateral grid after height processing, the first bilateral grid after width processing, and the first bilateral grid after depth processing to obtain a third bilateral grid; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral grid to obtain the second bilateral grid after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral grid to obtain the second bilateral grid after width processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid to obtain the second bilateral grid after depth processing; and performing weighted summation processing on the second bilateral grid after height processing, the second bilateral grid after width processing, and the second bilateral grid after depth processing to obtain a fourth bilateral grid.

[0035] In a possible implementation, performing mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and performing mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral mesh to obtain the first bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral mesh to obtain the first bilateral mesh after width processing; after obtaining the first bilateral mesh after height processing and the first bilateral mesh after width processing, weighted summation processing can be performed on these two bilateral meshes to obtain a third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral mesh to obtain the second bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral mesh to obtain the second bilateral mesh after width processing; performing weighted summation processing on the second bilateral mesh after height processing and the second bilateral mesh after width processing to obtain a fourth bilateral mesh.

[0036] In a possible implementation, performing mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and performing mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral mesh to obtain the first bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral mesh to obtain the first bilateral mesh after depth processing; after obtaining the first bilateral mesh after height processing and the first bilateral mesh after depth processing, weighted summation processing can be performed on these two bilateral meshes to obtain a third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral mesh to obtain the second bilateral mesh after height processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral mesh to obtain the second bilateral mesh after depth processing; performing weighted summation processing on the second bilateral mesh after height processing and the second bilateral mesh after depth processing to obtain a fourth bilateral mesh.

[0037] In a possible implementation, performing mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and performing mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral mesh to obtain the first bilateral mesh after width processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral mesh to obtain the first bilateral mesh after depth processing; after obtaining the first bilateral mesh after width processing and the first bilateral mesh after depth processing, weighted summation processing can be performed on these two bilateral meshes to obtain a third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral mesh to obtain the second bilateral mesh after width processing; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral mesh to obtain the second bilateral mesh after depth processing; performing weighted summation processing on the second bilateral mesh after width processing and the second bilateral mesh after depth processing to obtain a fourth bilateral mesh.

[0038] In a possible implementation, performing mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and performing mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral mesh to obtain a third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral mesh to obtain a fourth bilateral mesh.

[0039] In a possible implementation, performing mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and performing mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral mesh to obtain a third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral mesh to obtain a fourth bilateral mesh.

[0040] In a possible implementation, performing mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and performing mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh includes: respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral mesh to obtain a third bilateral mesh; respectively performing Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral mesh to obtain a fourth bilateral mesh.

[0041] In a possible implementation, the model to be trained further includes a feature extraction layer, which performs upsampling on the third bilateral grid to obtain a fifth bilateral grid, and performs upsampling on the fourth bilateral grid to obtain a sixth bilateral grid, including: performing feature extraction on the image to be processed through the feature extraction layer to obtain a texture guidance map and a color guidance map; performing interpolation on the third bilateral grid according to the texture guidance map to obtain a fifth bilateral grid, and performing interpolation on the fourth bilateral grid according to the color guidance map to obtain a sixth bilateral grid.

[0042] In a possible implementation, performing image reconstruction on the fifth feature and the sixth feature through an image reconstruction layer to obtain a predicted image includes: performing a third fusion process on the fifth feature and the sixth feature through the image reconstruction layer to obtain a seventh feature; performing convolution on the seventh feature through the image reconstruction layer to obtain a predicted image.

[0043] In a possible implementation, the third fusion process includes at least one of normalization, summation, multiplication, concatenation, and concatenation convolution.

[0044] The third aspect of the embodiments of the present application provides an image enhancement device, including: a first acquisition module, configured to acquire a first feature of the texture information of the target image and a second feature of the color information of the target image through an image processing model; a fusion module, configured to perform a first fusion process on the first feature and the second feature through the image processing model to obtain a third feature of the texture information, and perform a second fusion process on the first feature and the second feature through the image processing model to obtain a fourth feature of the color information; a generation module, configured to generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature; an enhancement module, configured to perform enhancement processing on the target image according to the first bilateral grid and the second bilateral network to obtain an enhanced target image.

[0045] It can be seen from the above device that: since the third feature is obtained by integrating the second feature of the color information on the basis of the first feature of the texture information, and the fourth feature is obtained by integrating the first feature of the texture information on the basis of the second feature of the color information, the first bilateral grid generated based on the third feature not only contains the texture information of the target image, but also contains the correlation between the texture information and the color information. The second bilateral grid generated based on the fourth feature not only contains the color information of the target image, but also contains the correlation between the texture information and the color information. The information about the target image contained in the two bilateral grids is relatively comprehensive. Then, the enhanced target image obtained based on the first bilateral grid and the second bilateral grid can accurately display the image information that needs to be highlighted, thereby improving the image enhancement effect.

[0046] In a possible implementation, the image processing model includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. The first acquisition module is configured to: perform convolutional processing on the texture information of the target image through the N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; perform convolutional processing on the color information of the target image through the N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

[0047] In a possible implementation, the image processing model further includes N first fusion layers and N second fusion layers. The fusion module is configured to: perform convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; perform normalization processing on the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; perform convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1, …, N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

[0048] In a possible implementation, the apparatus further includes: a second acquisition module, configured to acquire a target image; a downsampling module, configured to perform downsampling processing on the target image to obtain a downsampled target image; a separation module, configured to perform separation processing on the downsampled target image to obtain the texture information of the target image and the color information of the target image.

[0049] In a possible implementation, the image processing model further includes an image reconstruction layer. The enhancement module is configured to: perform grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and perform grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid, where the grid reconstruction processing is used to remove some information in the bilateral grid; perform upsampling processing on the third bilateral grid to obtain a fifth bilateral grid, and perform upsampling processing on the fourth bilateral grid to obtain a sixth bilateral grid; perform affine transformation on the target image according to the fifth bilateral grid to obtain a fifth feature, and perform affine transformation on the target image according to the sixth bilateral grid to obtain a sixth feature; perform image reconstruction processing on the fifth feature and the sixth feature through the image reconstruction layer to obtain an enhanced target image.

[0050] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with the depth processed; perform weighted summation on the first bilateral grid with the height processed, the first bilateral grid with the width processed, and the first bilateral grid with the depth processed to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed; perform weighted summation on the second bilateral grid with the height processed, the second bilateral grid with the width processed, and the second bilateral grid with the depth processed to obtain a fourth bilateral grid.

[0051] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed; after obtaining the first bilateral grid with the height processed and the first bilateral grid with the width processed, weighted summation can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed; perform weighted summation on the second bilateral grid with the height processed and the second bilateral grid with the width processed to obtain a fourth bilateral grid.

[0052] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with the depth processed; after obtaining the first bilateral grid with the height processed and the first bilateral grid with the depth processed, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed; perform weighted summation processing on the second bilateral grid with the height processed and the second bilateral grid with the depth processed to obtain a fourth bilateral grid.

[0053] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with the depth processed; after obtaining the first bilateral grid with the width processed and the first bilateral grid with the depth processed, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed; perform weighted summation processing on the second bilateral grid with the width processed and the second bilateral grid with the depth processed to obtain a fourth bilateral grid.

[0054] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0055] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0056] In a possible implementation, the enhancement module is configured to: perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid respectively to obtain a third bilateral grid; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0057] In a possible implementation, the image processing model further includes a feature extraction layer, and the enhancement module is configured to: perform feature extraction processing on the target image through the feature extraction layer to obtain a texture guidance map and a color guidance map; perform interpolation processing on the third bilateral grid according to the texture guidance map to obtain a fifth bilateral grid, and perform interpolation processing on the fourth bilateral grid according to the color guidance map to obtain a sixth bilateral grid.

[0058] In a possible implementation, the enhancement module is configured to: perform a third fusion process on the fifth feature and the sixth feature through the image reconstruction layer to obtain a seventh feature; perform convolution processing on the seventh feature through the image reconstruction layer to obtain the enhanced target image.

[0059] In a possible implementation, the third fusion process includes at least one of normalization processing, summation processing, multiplication processing, concatenation processing, and concatenated convolution processing.

[0060] The fourth aspect of the embodiments of the present application provides a model training device, which includes: a first acquisition module, configured to acquire a to-be-processed image and a real image after image enhancement of the to-be-processed image; a second acquisition module, configured to acquire a first feature of the texture information of the to-be-processed image and a second feature of the color information of the to-be-processed image through the to-be-trained model; a fusion module, configured to perform a first fusion process on the first feature and the second feature through the to-be-trained model to obtain a third feature of the texture information, and perform a second fusion process on the first feature and the second feature through the to-be-trained model to obtain a fourth feature of the color information; a generation module, configured to generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature; an enhancement module, configured to perform enhancement processing on the to-be-processed image according to the first bilateral grid and the second bilateral network to obtain a predicted image; a third acquisition module, configured to obtain a target loss according to the predicted image, the real image, and the processed predicted image, where the target loss is used to indicate the difference between the predicted image and the real image, and the difference between the predicted image and the processed predicted image; an update module, configured to update the model parameters of the to-be-trained model according to the target loss until the model training condition is met to obtain an image processing model.

[0061] The image processing model obtained based on the above device has the ability to enhance the image to be processed. During the image enhancement process, the image processing model can incorporate the second feature of color information based on the first feature of texture information to obtain a third feature, and the image processing model can incorporate the first feature of texture information based on the second feature of color information to obtain a fourth feature. Therefore, the first bilateral grid generated based on the third feature not only contains the texture information of the image to be processed but also the correlation between the texture information and the color information. The second bilateral grid generated based on the fourth feature not only contains the color information of the image to be processed but also the correlation between the texture information and the color information. The information about the image to be processed contained in the two bilateral grids is relatively comprehensive. Then, the enhanced image to be processed obtained based on the first bilateral grid and the second bilateral grid can accurately display the image information that needs to be highlighted, thereby improving the image enhancement effect.

[0062] In a possible implementation, the processed predicted image is the predicted image after pixel flipping and blurring processing.

[0063] In a possible implementation, the model to be trained includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. The second acquisition module is configured to: perform convolutional processing on the texture information of the image to be processed through the N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; perform convolutional processing on the color information of the image to be processed through the N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

[0064] In a possible implementation, the model to be trained further includes N first fusion layers and N second fusion layers. The fusion module is configured to: perform convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; perform normalization processing on the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; perform convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature, where i = 1, …, N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

[0065] In a possible implementation, the device further includes: a fourth acquisition module, configured to acquire an image to be processed; a downsampling module, configured to perform downsampling processing on the image to be processed to obtain a downsampled image to be processed; and a separation module, configured to perform separation processing on the downsampled image to be processed to obtain the texture information of the image to be processed and the color information of the image to be processed.

[0066] In a possible implementation, the model to be trained further includes an image reconstruction layer, and an enhancement module, configured to: perform mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and perform mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh, where the mesh reconstruction processing is used to remove some information in the bilateral mesh; perform upsampling processing on the third bilateral mesh to obtain a fifth bilateral mesh, and perform upsampling processing on the fourth bilateral mesh to obtain a sixth bilateral mesh; perform affine transformation on the image to be processed according to the fifth bilateral mesh to obtain a fifth feature, and perform affine transformation on the image to be processed according to the sixth bilateral mesh to obtain a sixth feature; and perform image reconstruction processing on the fifth feature and the sixth feature through the image reconstruction layer to obtain a predicted image.

[0067] In a possible implementation, the enhancement module is configured to: perform Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral mesh respectively to obtain a first bilateral mesh with processed height; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral mesh respectively to obtain a first bilateral mesh with processed width; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral mesh respectively to obtain a first bilateral mesh with processed depth; perform weighted summation processing on the first bilateral mesh with processed height, the first bilateral mesh with processed width, and the first bilateral mesh with processed depth to obtain a third bilateral mesh; perform Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral mesh respectively to obtain a second bilateral mesh with processed height; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral mesh respectively to obtain a second bilateral mesh with processed width; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral mesh respectively to obtain a second bilateral mesh with processed depth; perform weighted summation processing on the second bilateral mesh with processed height, the second bilateral mesh with processed width, and the second bilateral mesh with processed depth to obtain a fourth bilateral mesh.

[0068] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with processed height; perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with processed width; after obtaining the first bilateral grid with processed height and the first bilateral grid with processed width, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with processed height; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with processed width; perform weighted summation processing on the second bilateral grid with processed height and the second bilateral grid with processed width to obtain a fourth bilateral grid.

[0069] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with processed height; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with processed depth; after obtaining the first bilateral grid with processed height and the first bilateral grid with processed depth, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with processed height; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with processed depth; perform weighted summation processing on the second bilateral grid with processed height and the second bilateral grid with processed depth to obtain a fourth bilateral grid.

[0070] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid after width processing; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing; after obtaining the first bilateral grid after width processing and the first bilateral grid after depth processing, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid after width processing; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid after depth processing; perform weighted summation processing on the second bilateral grid after width processing and the second bilateral grid after depth processing to obtain a fourth bilateral grid.

[0071] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0072] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0073] In a possible implementation, an enhancement module is configured to: perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain a fourth bilateral grid.

[0074] In a possible implementation, the model to be trained further includes a feature extraction layer, and the enhancement module is configured to: perform feature extraction processing on the image to be processed through the feature extraction layer to obtain a texture guidance map and a color guidance map; perform interpolation processing on the third bilateral grid according to the texture guidance map to obtain a fifth bilateral grid, and perform interpolation processing on the fourth bilateral grid according to the color guidance map to obtain a sixth bilateral grid.

[0075] In a possible implementation, the enhancement module is configured to: perform a third fusion process on the fifth feature and the sixth feature through an image reconstruction layer to obtain a seventh feature; perform convolution processing on the seventh feature through the image reconstruction layer to obtain a predicted image.

[0076] In a possible implementation, the third fusion process includes at least one of a normalization process, a summation process, a multiplication process, a concatenation process, and a concatenated convolution process.

[0077] A fifth aspect of the embodiments of the present application provides an image enhancement device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the image enhancement device executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0078] A sixth aspect of the embodiments of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes the method described in the second aspect or any possible implementation manner of the second aspect.

[0079] A seventh aspect of the embodiments of the present application provides a circuit system, which includes a processing circuit configured to execute the method described in the first aspect, any possible implementation manner of the first aspect, the second aspect, or any possible implementation manner of the second aspect.

[0080] An eighth aspect of the embodiments of the present application provides a chip system, which includes a processor for calling a computer program or computer instruction stored in a memory, so that the processor executes the method described in the first aspect, any possible implementation manner of the first aspect, the second aspect, or any possible implementation manner of the second aspect.

[0081] In a possible implementation, the processor is coupled to the memory through an interface.

[0082] In a possible implementation, the chip system further includes a memory, and a computer program or computer instruction is stored in the memory.

[0083] A ninth aspect of the embodiments of the present application provides a computer storage medium, which stores a computer program. When the program is executed by a computer, the computer implements the method described in the first aspect, any possible implementation manner of the first aspect, the second aspect, or any possible implementation manner of the second aspect.

[0084] A tenth aspect of the embodiments of the present application provides a computer program product, which stores an instruction. When the instruction is executed by a computer, the computer implements the method described in the first aspect, any possible implementation manner of the first aspect, the second aspect, or any possible implementation manner of the second aspect.

[0085] In the embodiments of the present application, after obtaining the target image to be enhanced, the first feature of the texture information of the target image and the second feature of the color information of the target image can be obtained through the image processing model. Then, the first feature and the second feature are subjected to a first fusion process through the image processing model to obtain the third feature of the texture information, and the first feature and the second feature are subjected to a second fusion process through the image processing model to obtain the fourth feature of the color information. Then, a first bilateral grid is generated according to the third feature, and a second bilateral grid is generated according to the fourth feature. Finally, the target image is enhanced according to the first bilateral grid and the second bilateral network to obtain the enhanced target image. In the foregoing process, since the third feature is obtained by integrating the second feature of the color information on the basis of the first feature of the texture information, and the fourth feature is obtained by integrating the first feature of the texture information on the basis of the second feature of the color information, the first bilateral grid generated based on the third feature not only contains the texture information of the target image, but also contains the correlation between the texture information and the color information. The second bilateral grid generated based on the fourth feature not only contains the color information of the target image, but also contains the correlation between the texture information and the color information. The information about the target image contained in the two bilateral grids is relatively comprehensive. Then, the enhanced target image obtained based on the first bilateral grid and the second bilateral grid can accurately display the image information that needs to be highlighted, thereby improving the image enhancement effect. Description of the Drawings

[0086] Figure 1 It is a schematic structural diagram of a kind of artificial intelligence main framework;

[0087] Figure 2a It is a schematic structural diagram of an image processing system provided by an embodiment of the present application;

[0088] Figure 2b It is another schematic structural diagram of an image processing system provided by an embodiment of the present application;

[0089] Figure 2c It is a schematic diagram of a related device for image processing provided by an embodiment of the present application;

[0090] Figure 3 It is a schematic diagram of the architecture of system 100 provided by an embodiment of the present application;

[0091] Figure 4 It is a schematic flowchart of an image enhancement method provided by an embodiment of the present application;

[0092] Figure 5 It is a schematic structural diagram of an image processing model provided by an embodiment of the present application;

[0093] Figure 6A schematic diagram of the first feature extraction layer and the fusion layer provided by the embodiment of the present application;

[0094] Figure 7 A schematic diagram of the first fusion layer and the second fusion layer provided by the embodiment of the present application;

[0095] Figure 8 A flowchart of the model training method provided by the embodiment of the present application;

[0096] Figure 9 A schematic diagram of the image enhancement device provided by the embodiment of the present application;

[0097] Figure 10 A schematic diagram of the model training device provided by the embodiment of the present application;

[0098] Figure 11 A schematic diagram of the execution device provided by the embodiment of the present application;

[0099] Figure 12 A schematic diagram of the training device provided by the embodiment of the present application;

[0100] Figure 13 A schematic diagram of the chip provided by the embodiment of the present application. Detailed implementation manners

[0101] The embodiment of the present application provides an image enhancement method and related devices, which can accurately display the image information that needs to be highlighted in the enhanced target image, thereby improving the image enhancement effect.

[0102] Terms such as "first" and "second" in the description, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing when describing objects with the same attributes in the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0103] With the rapid development of computer technology, people use a large number of various images in daily life and production activities. In order to improve the image quality, image enhancement can be performed. Image enhancement refers to a processing method that highlights certain information in the image according to specific requirements and weakens or removes certain unnecessary information.

[0104] In the image enhancement method provided by the related art, after obtaining the target image to be enhanced, the features of the target image can be extracted through an image processing model, and a corresponding bilateral grid can be generated based on the features. Then, the target image can be enhanced by using the bilateral grid, so as to obtain the enhanced target image.

[0105] However, only generating the bilateral grid based on the features of the entire target image, the considered factors are relatively single, that is, the information about the target image incorporated in the bilateral grid is relatively one-sided. Therefore, for the enhanced target image obtained based on the bilateral grid, some image information that needs to be highlighted (such as texture, color, saturation, brightness, dynamic range, contrast, etc.) cannot be accurately displayed, resulting in poor image enhancement effect.

[0106] To solve the above problems, the present application provides an image enhancement method, which can be implemented by combining artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence. AI technology obtains the best results by perceiving the environment, acquiring knowledge, and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Using artificial intelligence for image processing is a common application method of artificial intelligence.

[0107] First, the overall working process of the artificial intelligence system is described. Please refer to Figure 1 , Figure 1 is a schematic structural diagram of the artificial intelligence main framework. The above artificial intelligence main framework is described from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes the refinement process of "data - information - knowledge - wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0108] (1) Infrastructure

[0109] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the external world, and is supported through the basic platform. It communicates with the external world through sensors; the computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); the basic platform includes relevant platform guarantees and supports such as distributed computing frameworks and networks, and can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to obtain data, and this data is provided to the intelligent chips in the distributed computing system provided by the basic platform for computing.

[0110] (2) Data

[0111] The data at the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voices, texts, and also involves the Internet of Things data of traditional devices, including the business data of existing systems and the sensed data such as force, displacement, liquid level, temperature, humidity, etc.

[0112] (3) Data Processing

[0113] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0114] Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on the data.

[0115] Reasoning refers to the process of simulating the intelligent reasoning method of humans in a computer or intelligent system, and using formal information to perform machine thinking and solve problems according to the reasoning control strategy. The typical function is search and matching.

[0116] Decision-making refers to the process of making decisions after the intelligent information is reasoned, and usually provides functions such as classification, sorting, prediction, etc.

[0117] (4) General Capabilities

[0118] After the data undergoes the above-mentioned data processing, some general capabilities can be formed further based on the results of the data processing. For example, it can be an algorithm or a general system. For example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0119] (5) Intelligent Products and Industry Applications

[0120] Intelligent products and industry applications refer to the products and applications of the artificial intelligence system in various fields. It is the encapsulation of the overall artificial intelligence solution, productizes the intelligent information decision-making, and realizes the landing application. Its application fields mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0121] Next, several application scenarios of the present application will be introduced.

[0122] Figure 2a FIG. 1 is a schematic structural diagram of an image processing system provided by an embodiment of the present application. The image processing system includes a user device and a data processing device. Among them, the user device includes intelligent terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiating end of image processing and, as the initiator of an image processing request, usually a user initiates a request through the user device.

[0123] The above data processing device can be a device or server with data processing functions such as a cloud server, a network server, an application server, and a management server. The data processing device receives an image processing request from the intelligent terminal through an interaction interface, and then performs image processing in ways such as machine learning, deep learning, searching, reasoning, and decision-making through a memory for storing data and a processor for data processing. The memory in the data processing device can be a general term, including local storage and a database for storing historical data. The database can be on the data processing device or on other network servers.

[0124] In Figure 2a the image processing system shown in FIG. 1, the user device can receive a user's instruction. For example, the user device can obtain an image input / selected by the user, and then send a request to the data processing device, so that the data processing device executes an image processing application (such as image enhancement, etc.) on the image obtained by the user device, thereby obtaining a corresponding processing result for the image. Exemplarily, the user device can obtain an image input by the user, and then send an image enhancement request to the data processing device, so that the data processing device performs image enhancement on the image, thereby obtaining an enhanced image.

[0125] In Figure 2a FIG. 1, the data processing device can execute the image processing method of the embodiment of the present application.

[0126] Figure 2b FIG. 2 is another schematic structural diagram of an image processing system provided by an embodiment of the present application. In Figure 2b FIG. 2, the user device directly serves as the data processing device. The user device can directly obtain an input from the user and directly process it by the hardware of the user device itself. The specific process is similar to Figure 2a that in FIG. 1, which can be referred to the above description and will not be elaborated here.

[0127] In Figure 2bIn the image processing system shown, the user device can receive user instructions. For example, the user device can obtain an image selected by the user in the user device, and then the user device itself can execute an image processing application (such as image enhancement, etc.) on the image, so as to obtain a corresponding processing result for the image.

[0128] In Figure 2b the user device itself can execute the image enhancement method of the embodiments of the present application.

[0129] Figure 2c FIG. is a schematic diagram of related devices for image processing provided by the embodiments of the present application.

[0130] The above Figure 2a and Figure 2b the user device in can specifically be Figure 2c the local device 301 or the local device 302 in, Figure 2a the data processing device in can specifically be Figure 2c the execution device 210 in. Among them, the data storage system 250 can store the data to be processed by the execution device 210. The data storage system 250 can be integrated on the execution device 210, or can be set on the cloud or other network servers.

[0131] Figure 2a and Figure 2b the processor in can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to execute an image processing application on the image, so as to obtain a corresponding processing result.

[0132] Figure 3 FIG. is a schematic diagram of the architecture of the system 100 provided by the embodiments of the present application. In Figure 3 the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. The user can input data to the I / O interface 112 through the client device 140. The input data in the embodiments of the present application can include: various tasks to be scheduled, callable resources, and other parameters.

[0133] During the preprocessing of the input data by the execution device 110, or during the process of the calculation module 111 of the execution device 110 performing calculations and other related processing (such as implementing the functions of the neural network in the present application), the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0134] Finally, the I / O interface 112 returns the processing result to the client device 140 for providing to the user.

[0135] It should be noted that the training device 120 can generate corresponding target models / rules based on different training data for different targets or tasks, and the corresponding target models / rules can be used to achieve the above-mentioned targets or complete the above-mentioned tasks, so as to provide the required results for the user. Among them, the training data can be stored in the database 130 and comes from the training samples collected by the data collection device 160.

[0136] In Figure 3 the shown case, the user can manually give the input data, and this manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send the input data to the I / O interface 112. If the client device 140 is required to automatically send the input data and user authorization is needed, the user can set the corresponding permissions in the client device 140. The user can view the results output by the execution device 110 in the client device 140, and the specific presentation form can be specific ways such as display, sound, action, etc. The client device 140 can also be used as a data collection end to collect the input data input to the I / O interface 112 and the output result output from the I / O interface 112 as new sample data and store them in the database 130. Of course, it can also be collected without passing through the client device 140, but directly by the I / O interface 112 to store the input data input to the I / O interface 112 and the output result output from the I / O interface 112 as new sample data in the database 130.

[0137] It should be noted that Figure 3 is only a schematic diagram of a system architecture provided by the embodiments of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. As Figure 3 shown, a neural network can be trained according to the training device 120.

[0138] The embodiments of the present application also provide a chip, which includes a neural network processor NPU. This chip can be set in the execution device 110 as Figure 3 shown to complete the calculation work of the calculation module 111. This chip can also be set in the training device 120 as Figure 3 shown to complete the training work of the training device 120 and output the target model / rule.

[0139] The neural network processor NPU is attached to the main central processing unit (CPU) (host CPU) as a coprocessor, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit, and the controller controls the arithmetic circuit to extract data from the memory (weight memory or input memory) and perform operations.

[0140] In some implementations, the arithmetic circuit includes multiple processing engines (PEs) inside. In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general matrix processor.

[0141] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory and performs matrix operations with matrix B, and the partial results or final results of the obtained matrix are stored in the accumulator.

[0142] The vector calculation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, etc. For example, the vector calculation unit can be used for network calculations in non-convolutional / non-FC layers of a neural network, such as pooling, batch normalization, local response normalization, etc.

[0143] In some implementations, the vector calculation unit can store the processed output vector in the unified buffer. For example, the vector calculation unit can apply a non-linear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit generates normalized values, combined values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit, such as for use in subsequent layers in a neural network.

[0144] The unified memory is used to store input data and output data.

[0145] The weight data directly transfers the input data in the external memory to the input memory and / or the unified memory, stores the weight data in the external memory into the weight memory, and stores the data in the unified memory into the external memory through the direct memory access controller (DMAC).

[0146] A bus interface unit (BIU) is used to enable interaction among the main CPU, DMAC, and the instruction fetch memory via a bus.

[0147] An instruction fetch buffer connected to the controller is used to store the instructions used by the controller;

[0148] The controller is used to call the instructions cached in the instruction fetch memory to control the operation process of the arithmetic accelerator.

[0149] Generally, the unified memory, input memory, weight memory, and instruction fetch memory are all on-chip memories, and the external memory is the memory outside the NPU. The external memory can be a double data rate synchronous dynamic random access memory (DDRSDRAM), a high bandwidth memory (HBM), or other readable and writable memories.

[0150] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0151] (1) Neural network

[0152] A neural network can be composed of neural units. A neural unit can be an arithmetic unit with xs and intercept 1 as inputs, and the output of this arithmetic unit can be:

[0153]

[0154] where s = 1, 2,... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many such single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0155] The operation of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). Physically, the operation of each layer in a neural network can be understood as completing the transformation from the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by Wx, operation 4 is completed by +b, and operation 5 is implemented by a(). The reason for using the word "space" here is that the objects to be classified are not individual things but a class of things. Space refers to the set of all individuals of this class of things. Among them, W is the weight vector, and each value in this vector represents the weight value of a neuron in this layer of the neural network. This vector W determines the space transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to finally obtain the weight matrices of all layers of the trained neural network (the weight matrix formed by vectors W of many layers). Therefore, the training process of a neural network is essentially a process of learning the way to control space transformation, and more specifically, learning the weight matrix.

[0156] Because it is hoped that the output of the neural network is as close as possible to the value that is really wanted to be predicted, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network and the value that is really wanted, and then according to the difference between the two. (Of course, there is usually an initialization process before the first update, that is, parameters are pre-configured for each layer in the neural network). For example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and continuous adjustment is made until the neural network can predict the value that is really wanted. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function. They are important equations for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the neural network becomes a process of minimizing this loss as much as possible.

[0157] (2) Backpropagation algorithm

[0158] The neural network can use the backpropagation (BP) algorithm to correct the magnitudes of the parameters in the initial neural network model during the training process, making the reconstruction error loss of the neural network model smaller and smaller. Specifically, forward propagating the input signal until the output generates an error loss, and updating the parameters in the initial neural network model by backpropagating the error loss information, so as to converge the error loss. The backpropagation algorithm is a backpropagation movement dominated by the error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0159] The method provided in this application will be described below from the training side of the neural network and the application side of the neural network.

[0160] The model training method provided in the embodiments of this application is related to image processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on training data (such as the target image in the model training method provided in the embodiments of this application), and finally obtains a trained neural network (such as the image processing model in the embodiments of this application); moreover, the image enhancement method provided in the embodiments of this application can use the above-trained neural network to input the input data (such as the target image in the image enhancement method provided in the embodiments of this application) into the trained neural network to obtain output data (such as in the image enhancement method provided in the embodiments of this application, the first feature of the texture information of the target image, the second feature of the color information of the target image, the texture guidance map, the color guidance map, the seventh feature, and the enhanced target image, etc.). It should be noted that the model training method and the image enhancement method provided in the embodiments of this application are inventions based on the same concept, and can also be understood as two parts of a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0161] Figure 4 FIG. is a schematic flowchart of an image enhancement method provided in the embodiments of this application. This method can be implemented through an image processing model and a mathematical transformation module. The relationship between the image processing model and the mathematical transformation module will be briefly introduced below in conjunction with Figure 5 As shown in Figure 5 ( Figure 5 is a schematic structural diagram of an image processing model provided in the embodiments of this application), the image processing model is divided into two parts, and the mathematical transformation module is arranged between the first part and the second part of the image processing model. Among them, the first part of the image processing model includes a first feature extraction layer, a second feature extraction layer, and a fusion layer, and the second part of the image processing module includes an image reconstruction layer. The following will be described in conjunction with Figure 4 and Figure 5, a specific introduction to the image enhancement method provided in this application is given. The method includes:

[0162] 401. Preprocess the target image to obtain the texture information and color information of the target image.

[0163] After obtaining the target image to be enhanced, the target image can be preprocessed first to obtain the texture information (texture image) and color information (color image) of the target image. Specifically, the preprocessing of the target image can be achieved in the following ways:

[0164] After obtaining the target image, first perform downsampling processing on the target image to obtain the downsampled target image. It can be understood that the downsampled target image has a smaller size and lower resolution compared to the original target image. Then, perform separation processing on the downsampled target image to obtain the texture information and color information of the target image. For example, a certain degree of Gaussian blur processing can be performed on the downsampled target image, and then the high-frequency information is removed to obtain the color information of the target image, and the texture information of the target image is obtained by subtracting the color information from the downsampled target image, and so on.

[0165] 402. Obtain the first feature of the texture information of the target image and the second feature of the color information of the target image through an image processing model.

[0166] 403. Perform a first fusion process on the first feature and the second feature through an image processing model to obtain the third feature of the texture information, and perform a second fusion process on the first feature and the second feature to obtain the fourth feature of the color information.

[0167] After obtaining the texture information and color information of the target image, the texture information and color information of the target image can be input into the first feature extraction layer of the image processing model, so as to perform feature extraction processing on the texture information of the target image through the first feature extraction layer to obtain the first feature of the texture information of the target image, and perform feature extraction processing on the color information of the target image through the first feature extraction layer to obtain the second feature of the color information of the target image. After the first feature extraction layer obtains the first feature and the second feature, the first feature and the second feature can be input into the fusion layer, so as to perform a first fusion process on the first feature and the second feature through the fusion layer to obtain the third feature of the texture information, and perform a second fusion process on the first feature and the second feature through the fusion layer to obtain the fourth feature of the color information.

[0168] Specifically, as Figure 6 shown ( Figure 6(Schematic diagram of the first feature extraction layer and the fusion layer provided by the embodiment of the present application). The first feature extraction layer generally includes N convolutional layers connected in sequence. The fusion layer includes N first fusion layers and N second fusion layers. Among them, the first first fusion layer is connected to the first convolutional layer, the i-th first fusion layer is connected to the i-th second fusion layer, and the i-th convolutional layer is connected to the i-th second fusion layer, where i = 1,..., N, and N is an integer greater than or equal to 1.

[0169] After obtaining the texture information and color information of the target image, they can be input into the first convolutional layer among the N convolutional layers, so as to perform multi-level convolutional processing on the texture information of the target image through the N convolutional layers, and obtain the first sub-features respectively output by the N convolutional layers, that is, N first sub-features (equivalent to the first feature mentioned above), and perform multi-level convolutional processing on the color information of the target image through the N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers, that is, N second sub-features (equivalent to the second feature mentioned above).

[0170] After obtaining the N first sub-features and the N second sub-features, the i-th first fusion layer can perform convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer to obtain the i-th third sub-feature, and the i-th second fusion layer can perform normalization processing on the i-th first sub-feature and the i-th second sub-feature to obtain the i-th fifth sub-feature, and then the i-th first fusion layer can perform convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature to obtain the i-th fourth sub-feature. Where i = 1,..., N. When i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature.

[0171] For example, the first convolutional layer can input the first first sub-feature and the first second sub-feature into the first first fusion layer and the first second fusion layer, so as to perform convolutional processing and normalization processing on the first first sub-feature and the first second sub-feature through the first first fusion layer to obtain the first third sub-feature, and perform normalization processing on the first first sub-feature and the first second sub-feature through the first second fusion layer to obtain the first fifth sub-feature, and then perform convolutional processing and addition processing on the first second sub-feature and the first fifth sub-feature through the first first fusion layer to obtain the first fourth sub-feature.

[0172] The first first fusion layer can input the first third sub - feature and the first fourth sub - feature into the second first fusion layer, and the second convolutional layer can input the second first sub - feature and the second second sub - feature into the second second fusion layer, so as to perform convolutional processing and normalization processing on the first third sub - feature and the first fourth sub - feature through the second first fusion layer to obtain the second third sub - feature, and perform normalization processing on the second first sub - feature and the second second sub - feature through the second second fusion layer to obtain the second fifth sub - feature, and then perform convolutional processing and addition processing on the first fourth sub - feature and the second fifth sub - feature through the second first fusion layer to obtain the second fourth sub - feature.

[0173] And so on, the (N - 1)th first fusion layer can input the (N - 1)th third sub - feature and the (N - 1)th fourth sub - feature into the Nth first fusion layer, and the Nth convolutional layer can input the Nth first sub - feature and the Nth second sub - feature into the Nth second fusion layer, so as to perform convolutional processing and normalization processing on the (N - 1)th third sub - feature and the (N - 1)th fourth sub - feature through the Nth first fusion layer to obtain the Nth third sub - feature, and perform normalization processing on the Nth first sub - feature and the Nth second sub - feature through the Nth second fusion layer to obtain the Nth fifth sub - feature, and then perform convolutional processing and addition processing on the (N - 1)th fourth sub - feature and the Nth fifth sub - feature through the Nth first fusion layer to obtain the Nth fourth sub - feature.

[0174] Further, the structure of the first fusion layer can be as Figure 7 shown ( Figure 7 which is a schematic structural diagram of the first fusion layer and the second fusion layer provided by the embodiment of the present application). The first fusion layer can include two branches. The first branch can include a convolutional layer, a normalization layer, and a convolutional layer connected directly in sequence. The second branch can include a convolutional layer, an addition layer, and a convolutional layer connected directly in sequence. The second fusion layer is a normalization layer. It should be noted that the normalization layer included in the first fusion layer and the second fusion layer is the same. To further understand the working process of the first fusion layer and the second fusion layer, the working process of the Nth first fusion layer and the Nth second fusion layer is introduced below:

[0175] After the (N - 1)th first fusion layer sends the (N - 1)th third sub - feature to the first branch of the Nth first fusion layer, the convolutional layer received by the first branch can perform convolutional processing on the (N - 1)th third sub - feature to obtain the convolved (N - 1)th third sub - feature. Similarly, after the (N - 1)th first fusion layer sends the (N - 1)th fourth sub - feature to the second branch of the Nth first fusion layer, the convolutional layer of the second branch can perform convolutional processing on the (N - 1)th fourth sub - feature to obtain the convolved (N - 1)th fourth sub - feature.

[0176] Meanwhile, after the Nth convolutional layer sends the Nth first sub-feature and the Nth second sub-feature to the Nth second fusion layer, the Nth second fusion layer normalizes the Nth first sub-feature and the Nth second sub-feature to obtain the Nth fifth sub-feature.

[0177] Next, the normalization layer of the first branch can normalize the N-1th third sub-feature after convolution and the N-1th fourth sub-feature after convolution to obtain the normalized feature. Then, the convolutional layer of the first branch convolves the normalized feature to obtain the Nth third sub-feature.

[0178] Correspondingly, the addition layer of the second branch can add the N-1th fourth sub-feature after convolution and the Nth fifth sub-feature to obtain the added feature. Then, the convolutional layer of the second branch convolves the added feature to obtain the Nth fourth sub-feature.

[0179] It should be noted that the normalization process performed by the normalization layer of the Nth first fusion layer and the normalization process performed by the Nth second fusion layer are the same, and the normalization processes of the two can be achieved through the following formula:

[0180]

[0181] In the above formula, t is the texture feature (for example, the Nth first sub-feature and the N-1th third sub-feature after convolution), c is the color feature (for example, the Nth second sub-feature and the N-1th fourth sub-feature after convolution), sigmoid is the non-linear sigmoid function, pool is the pooling process, μ is the average calculation process, and σ is the standard deviation calculation process.

[0182] After obtaining the Nth third sub-feature and the Nth fourth sub-feature, it is equivalent to obtaining the third feature of the texture information and the fourth feature of the color information.

[0183] It should be understood that in the N convolutional layers, two adjacent convolutional layers can be directly connected or indirectly connected (that is, several convolutional layers can be arranged between two adjacent convolutional layers), which is not limited here.

[0184] It should also be understood that for the convenience of introduction, in this embodiment, only the first feature extraction layer including N convolutional layers is used for illustrative description. In practical applications, the first feature extraction layer usually includes two branches. The first branch includes N first convolutional layers connected in sequence, which can be used for multi-level convolution processing of the texture information of the target image. The second branch also includes N second convolutional layers connected in sequence, which can be used for multi-level convolution processing of the color information of the target image. It can be seen that in this embodiment, each convolutional layer in the N convolutional layers includes a first convolutional layer and a second convolutional layer.

[0185] It should also be understood that since the N convolutional layers are connected in sequence, among the N first sub-features, the second first sub-feature is determined based on the first first sub-feature, the third first sub-feature is determined based on the second first sub-feature, ..., and the Nth first sub-feature is determined based on the (N - 1)th first sub-feature. Similarly, the same applies to the N second sub-features, which will not be elaborated here.

[0186] It should also be understood that Figure 7 It only serves as a schematic illustration and does not limit the structure of the first fusion layer of the present application. For example, the first branch of the first fusion layer may include three convolutional layers and two normalization layers, and the second branch may include three convolutional layers, etc.

[0187] It should also be understood that the working processes of the first first fusion layer and the first second fusion layer can refer to the working processes of the Nth first fusion layer and the Nth second fusion layer. Similarly, the working processes of the second first fusion layer and the second second fusion layer can refer to the working processes of the Nth first fusion layer and the Nth second fusion layer, ..., and the working processes of the (N - 1)th first fusion layer and the (N - 1)th second fusion layer can refer to the working processes of the Nth first fusion layer and the Nth second fusion layer.

[0188] 404. Generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature.

[0189] After obtaining the third feature of the texture information and the fourth feature of the color information, a first bilateral grid of the texture information can be generated according to the third feature, and a second bilateral grid of the color information can be generated according to the fourth feature. It should be noted that the third feature and the fourth feature usually have three-dimensional features. Then, the first bilateral grid generated based on the third feature is also a three-dimensional bilateral grid, and each dimension contains multiple affine transformation matrices. For example, assume there is a third feature with a size of [16, 16, 96], that is, the height of the third feature is 16, the width is 16, and the depth is 96. Then, a first bilateral grid with a size of [16, 16, 8, 12] can be generated based on the third feature. The height of the first bilateral grid is 16, the width is 16, the depth is 8, and each dimension contains an affine transformation matrix with a size of 3 * 4, that is, the first bilateral grid contains a total of 16 * 16 * 8 affine transformation matrices with a size of 3 * 4.

[0190] 405. Perform grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and perform grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid. The grid reconstruction processing is used to remove some information in the bilateral grid.

[0191] The first bilateral grid for obtaining texture information and the second bilateral grid for obtaining color information can respectively perform network reconstruction processing on the first bilateral grid and the second bilateral grid, and correspondingly obtain the third bilateral grid for texture information and the fourth bilateral grid for color information. Among them, the network reconstruction processing is used to remove the information with less influence in the bilateral grid and retain the information with greater influence (i.e., the core information) in the bilateral grid. Specifically, the foregoing network reconstruction processing can be implemented in multiple ways, which will be introduced separately below:

[0192] In a possible implementation, the height, width, and depth of the first bilateral grid can be decomposed and reconstructed, and then the three obtained bilateral grids are fused to obtain the third bilateral grid. Specifically, this process includes: (1) performing Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid after height processing. Still as in the above example, the height of the first bilateral grid is 16. The height of the first bilateral grid can be compressed from 16 to 6 (for the height of the first bilateral grid, it is equivalent to removing the less influential information and only retaining the more influential information), and then the height of the first bilateral grid is reconstructed from 6 to 16 (during the height reconstruction process, the previously retained more influential information remains unchanged and no new information is added). Thus, the Tucker decomposition and Tucker reconstruction of the height of the first bilateral grid are completed, and the first bilateral grid after height processing is obtained. (2) Performing Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid after width processing. Still as in the above example, the width of the first bilateral grid is 16. The width of the first bilateral grid can be compressed from 16 to 8 (for the width of the first bilateral grid, it is equivalent to removing the less influential information and only retaining the more influential information), and then the width of the first bilateral grid is reconstructed from 8 to 16 (during the width reconstruction process, the previously retained more influential information remains unchanged and no new information is added). Thus, the Tucker decomposition and Tucker reconstruction of the width of the first bilateral grid are completed, and the first bilateral grid after width processing is obtained. (3) Performing Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing. Still as in the above example, the depth of the first bilateral grid is 8. The depth of the first bilateral grid can be compressed from 8 to 2 (for the depth of the first bilateral grid, it is equivalent to removing the less influential information and only retaining the more influential information), and then the depth of the first bilateral grid is reconstructed from 2 to 8 (during the depth reconstruction process, the previously retained more influential information remains unchanged and no new information is added). Thus, the Tucker decomposition and Tucker reconstruction of the depth of the first bilateral grid are completed, and the first bilateral grid after depth processing is obtained. (4) After obtaining the first bilateral grid after height processing, the first bilateral grid after width processing, and the first bilateral grid after depth processing, weighted summation processing can be performed on these three bilateral grids to obtain the third bilateral grid.

[0193] Further, the height, width, and depth of the second bilateral grid can be decomposed and reconstructed, and then the three obtained bilateral grids can be fused to obtain a fourth bilateral grid. Specifically, this process includes: (1) Performing Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed. (2) Performing Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed. (3) Performing Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed. (4) Performing weighted summation on the second bilateral grid with the height processed, the second bilateral grid with the width processed, and the second bilateral grid with the depth processed to obtain the fourth bilateral grid. For the description of the Tucker decomposition and Tucker reconstruction of the height, width, and depth of the second bilateral grid, reference can be made to the relevant description part of the Tucker decomposition and Tucker reconstruction of the height, width, and depth of the aforementioned first bilateral grid, which will not be elaborated here.

[0194] In another possible implementation, the height and width of the first bilateral grid can be decomposed and reconstructed, and then the two obtained bilateral grids can be fused to obtain a third bilateral grid. Specifically, this process includes: (1) Performing Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed. (2) Performing Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed. (3) After obtaining the first bilateral grid with the height processed and the first bilateral grid with the width processed, weighted summation can be performed on these two bilateral grids to obtain the third bilateral grid.

[0195] Further, the height and width of the second bilateral grid can be decomposed and reconstructed, and then the two obtained bilateral grids can be fused to obtain a fourth bilateral grid. Specifically, this process includes: (1) Performing Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed. (2) Performing Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed. (3) Performing weighted summation on the second bilateral grid with the height processed and the second bilateral grid with the width processed to obtain the fourth bilateral grid.

[0196] In another possible implementation, the height and depth of the first bilateral grid can be decomposed and reconstructed, and then the two obtained bilateral grids can be fused to obtain a third bilateral grid. Specifically, this process includes: (1) performing Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid after height processing. (2) Performing Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing. (3) After obtaining the first bilateral grid after height processing and the first bilateral grid after depth processing, these two bilateral grids can be subjected to weighted summation processing to obtain a third bilateral grid.

[0197] Furthermore, the height and depth of the second bilateral grid can be decomposed and reconstructed, and then the two obtained bilateral grids can be fused to obtain a fourth bilateral grid. Specifically, this process includes: (1) performing Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid after height processing. (2) Performing Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid after depth processing. (3) Performing weighted summation processing on the second bilateral grid after height processing and the second bilateral grid after depth processing to obtain a fourth bilateral grid.

[0198] In another possible implementation, the width and depth of the first bilateral grid can be decomposed and reconstructed, and then the two obtained bilateral grids can be fused to obtain a third bilateral grid. Specifically, this process includes: (1) performing Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid after width processing. (2) Performing Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing. (3) After obtaining the first bilateral grid after width processing and the first bilateral grid after depth processing, these two bilateral grids can be subjected to weighted summation processing to obtain a third bilateral grid.

[0199] Further, the width and depth of the second bilateral grid can be decomposed and reconstructed, and then the two obtained bilateral grids are fused to obtain the fourth bilateral grid. Specifically, this process includes: (1) performing Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid after width processing. (2) Performing Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid after depth processing. (3) Performing weighted summation on the second bilateral grid after width processing and the second bilateral grid after depth processing to obtain the fourth bilateral grid.

[0200] In another possible implementation, the height of the first bilateral grid can be decomposed and reconstructed to obtain the third bilateral grid. Specifically, this process includes: performing Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the third bilateral grid.

[0201] Further, the height of the second bilateral grid can be decomposed and reconstructed to obtain the fourth bilateral grid. Specifically, this process includes: performing Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0202] In another possible implementation, the width of the first bilateral grid can be decomposed and reconstructed to obtain the third bilateral grid. Specifically, this process includes: performing Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the third bilateral grid.

[0203] Further, the width of the second bilateral grid can be decomposed and reconstructed to obtain the fourth bilateral grid. Specifically, this process includes: performing Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0204] In another possible implementation, the depth of the first bilateral grid can be decomposed and reconstructed to obtain the third bilateral grid. Specifically, this process includes: performing Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the third bilateral grid.

[0205] Further, the depth of the second bilateral grid can be decomposed and reconstructed to obtain the fourth bilateral grid. Specifically, this process includes: performing Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0206] 406. Upsample the third bilateral grid to obtain the fifth bilateral grid, and upsample the fourth bilateral grid to obtain the sixth bilateral grid.

[0207] The third bilateral grid with texture information and the fourth bilateral grid with color information can be respectively upsampled to obtain the fifth bilateral grid with texture information and the sixth bilateral grid with color information. Specifically, the upsampling process can be achieved in the following ways:

[0208] (1) After obtaining the target image, the target image can be input into the second feature extraction layer of the image processing model (for example, the second feature extraction layer can include at least one convolutional layer, etc.) to perform feature extraction processing (such as convolutional processing, etc.) on the target image through the second feature extraction layer, and obtain a texture guidance map and a color guidance map.

[0209] (2) After obtaining the third bilateral grid and the fourth bilateral grid, interpolation processing can be performed on the third bilateral grid with reference to the texture guidance map to obtain the fifth bilateral grid, and interpolation processing can be performed on the fourth bilateral grid with reference to the color guidance map to obtain the sixth bilateral grid. At this point, the resolution of the fifth bilateral grid is the same as that of the original target image, and the resolution of the sixth bilateral grid is also the same as that of the original target image.

[0210] It should be understood that Figure 5 For the image processing model shown, the second feature extraction layer can include two branches (such as Figure 5 the upper branch and the lower branch in

[0211] ). The structures of these two branches can be the same, but the parameters of the two branches are different. Therefore, one branch is used to perform feature extraction on the original target image to obtain the texture guidance map, and the other branch is used to perform feature extraction on the original target image to obtain the color guidance map.

[0212] After obtaining the fifth bilateral grid with texture information and the sixth bilateral grid with color information, the target image can be affine-transformed using the fifth bilateral grid to obtain the fifth feature of the texture information of the target image, and the target image can be affine-transformed according to the sixth bilateral grid to obtain the sixth feature of the color information of the target image. It can be understood that the resolution of the fifth feature is the same as that of the original target image, and the resolution of the sixth feature is also the same as that of the original target image.

[0213] 408. Use an image processing model to perform image reconstruction processing on the fifth feature and the sixth feature to obtain an enhanced target image.

[0214] After obtaining the fifth feature of the texture information of the target image and the sixth feature of the color information of the target image, the fifth feature and the sixth feature can be input into the image reconstruction layer of the image processing model, so that the image reconstruction layer performs image reconstruction processing on the fifth feature and the sixth feature to obtain an enhanced target image. Specifically, the aforementioned image reconstruction processing can be implemented in the following ways:

[0215] (1) Perform a third fusion process on the fifth feature and the sixth feature through the image reconstruction layer to obtain a seventh feature, which contains the texture information and color information of the target image. Among them, the third fusion process includes at least one of normalization processing, summation processing, multiplication processing, concatenation processing, and concatenated convolution processing. For example, the image reconstruction layer includes a normalization layer (this normalization layer can be the same as the normalization layer in the first fusion layer and the second fusion layer). After receiving the fifth feature and the sixth feature, the normalization layer performs normalization processing on the fifth feature and the sixth feature to obtain the seventh feature.

[0216] (2) After obtaining the seventh feature, perform convolution processing on the seventh feature through the image reconstruction layer to obtain an enhanced target image. Still as in the above example, the image reconstruction layer also includes a convolution layer connected in sequence. After receiving the seventh feature from the normalization layer, the convolution layer can perform convolution processing on the seventh feature to obtain an enhanced target image.

[0217] It should be understood that in this embodiment, the sub-steps (2) of steps 402, 403, and 406 and step 408 are all implemented through the image processing model, and steps 404, 405, the sub-step (1) of step 406, and step 407 are all implemented through the mathematical transformation module.

[0218] In addition, the image enhancement method provided in the embodiments of the present application can be compared with the image enhancement methods in the related art. Specifically, the same test samples can be used to test the image enhancement method provided in the embodiments of the present application and the image enhancement methods in the related art, and the results are shown in Table 1:

[0219] Table 1

[0220]

[0221]

[0222] As shown in Table 1, the method provided by the embodiments of the present application has better peak signal to noise ratio (PSNR), structural similarity (SSIM), quality score (Q-Score), and execution time compared to the methods of the related art. It can be seen that the performance of the method provided by the embodiments of the present application is significantly better than that of the methods of the related art.

[0223] In the embodiments of the present application, after obtaining the target image to be enhanced, the first feature of the texture information of the target image and the second feature of the color information of the target image can be obtained through the image processing model. Then, the first feature and the second feature are subjected to a first fusion process through the image processing model to obtain the third feature of the texture information, and the first feature and the second feature are subjected to a second fusion process through the image processing model to obtain the fourth feature of the color information. Then, a first bilateral grid is generated according to the third feature, and a second bilateral grid is generated according to the fourth feature. Finally, the target image is enhanced according to the first bilateral grid and the second bilateral network to obtain the enhanced target image. In the foregoing process, since the third feature is obtained by integrating the second feature of the color information on the basis of the first feature of the texture information, and the fourth feature is obtained by integrating the first feature of the texture information on the basis of the second feature of the color information, the first bilateral grid generated based on the third feature not only contains the texture information of the target image, but also contains the correlation relationship between the texture information and the color information. The second bilateral grid generated based on the fourth feature not only contains the color information of the target image, but also contains the correlation relationship between the texture information and the color information. The information about the target image contained in the two bilateral grids is relatively comprehensive. Then, the enhanced target image obtained based on the first bilateral grid and the second bilateral grid can accurately display the image information to be highlighted (for example, texture, color, saturation, brightness, dynamic range, contrast, etc.), thereby improving the image enhancement effect.

[0224] Further, in the embodiments of the present application, the first bilateral grid and the second bilateral grid can be respectively subjected to Tucker decomposition and Tucker reconstruction, which is beneficial to filtering out some less useful information in these two bilateral grids. Then, in the process of subsequent processing of the target image based on the obtained third bilateral grid and fourth bilateral grid, the problems such as noise and incorrect boundaries generated by the bilateral grid on the target image can be avoided.

[0225] Further, in the embodiments of the present application, most steps are performed based on the downsampled target image (for example, steps 401 to 405). Since the resolution of the downsampled target image is lower than that of the original target image, excessive computational and storage overheads will not be introduced. Therefore, the method provided by the embodiments of the present application can be implemented on electronic devices with relatively ordinary hardware resources (such as terminal devices like mobile phones).

[0226] The above is a detailed description of the image enhancement method provided by the embodiments of the present application. The following will introduce the model training method provided by the embodiments of the present application. Figure 8 is a schematic flowchart of the model training method provided by the embodiments of the present application, as Figure 8 shown, the method includes:

[0227] 801. Obtain the image to be processed and the real image after image enhancement of the image to be processed.

[0228] 802. Preprocess the image to be processed to obtain the texture information and color information of the image to be processed.

[0229] When it is necessary to train the model to be trained, a batch of training images can be obtained, including the images to be processed for training. Among them, the model to be trained includes two parts. The first part of the image processing model includes a first feature extraction layer, a second feature extraction layer, and a fusion layer, and the second part of the image processing module includes an image reconstruction layer. It should be noted that the real image after image enhancement of the image to be processed is determined.

[0230] For the image to be processed, it can be preprocessed to obtain the texture information and color information of the image to be processed. For the introduction of preprocessing, reference can be made to Figure 4 the relevant description part of step 401 in the embodiment shown, which will not be elaborated here.

[0231] 803. Obtain the first feature of the texture information of the image to be processed and the second feature of the color information of the image to be processed through the model to be trained.

[0232] After obtaining the texture information and color information of the image to be processed, the texture information and color information of the image to be processed can be input into the first feature extraction layer of the model to be trained, so as to perform feature extraction processing on the texture information through the first feature extraction layer to obtain the first feature of the texture information of the image to be processed, and perform feature extraction processing on the color information through the first feature extraction layer to obtain the second feature of the color information of the image to be processed.

[0233] Specifically, the first feature extraction layer includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. The first feature extraction layer can perform feature extraction processing in the following way:

[0234] Perform convolutional processing on the texture information of the image to be processed through N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers, and perform convolutional processing on the color information of the image to be processed through N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

[0235] 804. Perform a first fusion process on the first feature and the second feature through the model to be trained to obtain a third feature of the texture information, and perform a second fusion process on the first feature and the second feature to obtain a fourth feature of the color information.

[0236] After the first feature extraction layer obtains the first feature and the second feature, it can input the first feature and the second feature into the fusion layer, so that the fusion layer performs a first fusion process on the first feature and the second feature to obtain a third feature of the texture information, and performs a second fusion process on the first feature and the second feature through the fusion layer to obtain a fourth feature of the color information.

[0237] Specifically, the fusion layer includes N first fusion layers and N second fusion layers. The fusion layer can perform the first fusion process and the second fusion process in the following way:

[0238] Perform convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature, and perform normalization processing on the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature. Then, perform convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature.

[0239] Where i = 1, …, N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

[0240] After obtaining the N-th third sub-feature and the N-th fourth sub-feature, it is equivalent to obtaining the third feature of the texture information and the fourth feature of the color information.

[0241] 805. Generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature.

[0242] After obtaining the third feature of the texture information and the fourth feature of the color information, the first bilateral grid of the texture information can be generated according to the third feature, and the second bilateral grid of the color information can be generated according to the fourth feature.

[0243] 806. Perform a mesh reconstruction process on the first bilateral grid to obtain a third bilateral grid, and perform a mesh reconstruction process on the second bilateral grid to obtain a fourth bilateral grid. The mesh reconstruction process is used to remove some information in the bilateral grid.

[0244] After obtaining the first bilateral grid of the texture information and the second bilateral grid of the color information, the mesh reconstruction process can be respectively performed on the first bilateral grid and the second bilateral grid, and the third bilateral grid of the texture information and the fourth bilateral grid of the color information can be correspondingly obtained. Among them, the mesh reconstruction process is used to remove the information with less effect in the bilateral grid and retain the information with greater effect in the bilateral grid.

[0245] Specifically, the mesh reconstruction process can be implemented in various ways, which will be introduced separately below:

[0246] In a possible implementation manner, perform Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral grid respectively to obtain the first bilateral grid after height processing; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral grid respectively to obtain the first bilateral grid after width processing; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing; perform weighted summation processing on the first bilateral grid after height processing, the first bilateral grid after width processing, and the first bilateral grid after depth processing to obtain a third bilateral grid; perform Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral grid respectively to obtain the second bilateral grid after height processing; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral grid respectively to obtain the second bilateral grid after width processing; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid respectively to obtain the second bilateral grid after depth processing; perform weighted summation processing on the second bilateral grid after height processing, the second bilateral grid after width processing, and the second bilateral grid after depth processing to obtain a fourth bilateral grid.

[0247] In another possible implementation, the height of the first bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the first bilateral grid after height processing; the width of the first bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the first bilateral grid after width processing; after obtaining the first bilateral grid after height processing and the first bilateral grid after width processing, these two bilateral grids can be subjected to weighted summation processing to obtain the third bilateral grid; the height of the second bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the second bilateral grid after height processing; the width of the second bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the second bilateral grid after width processing; the second bilateral grid after height processing and the second bilateral grid after width processing are subjected to weighted summation processing to obtain the fourth bilateral grid.

[0248] In another possible implementation, the height of the first bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the first bilateral grid after height processing; the depth of the first bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the first bilateral grid after depth processing; after obtaining the first bilateral grid after height processing and the first bilateral grid after depth processing, these two bilateral grids can be subjected to weighted summation processing to obtain the third bilateral grid; the height of the second bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the second bilateral grid after height processing; the depth of the second bilateral grid is respectively subjected to Tucker decomposition processing and Tucker reconstruction processing to obtain the second bilateral grid after depth processing; the second bilateral grid after height processing and the second bilateral grid after depth processing are subjected to weighted summation processing to obtain the fourth bilateral grid.

[0249] In another possible implementation, Tucker decomposition and Tucker reconstruction are respectively performed on the width of the first bilateral grid to obtain the first bilateral grid after width processing; Tucker decomposition and Tucker reconstruction are respectively performed on the depth of the first bilateral grid to obtain the first bilateral grid after depth processing; after obtaining the first bilateral grid after width processing and the first bilateral grid after depth processing, weighted summation processing can be performed on these two bilateral grids to obtain the third bilateral grid; Tucker decomposition and Tucker reconstruction are respectively performed on the width of the second bilateral grid to obtain the second bilateral grid after width processing; Tucker decomposition and Tucker reconstruction are respectively performed on the depth of the second bilateral grid to obtain the second bilateral grid after depth processing; weighted summation processing is performed on the second bilateral grid after width processing and the second bilateral grid after depth processing to obtain the fourth bilateral grid.

[0250] In another possible implementation, Tucker decomposition and Tucker reconstruction are respectively performed on the height of the first bilateral grid to obtain the third bilateral grid; Tucker decomposition and Tucker reconstruction are respectively performed on the height of the second bilateral grid to obtain the fourth bilateral grid.

[0251] In another possible implementation, Tucker decomposition and Tucker reconstruction are respectively performed on the width of the first bilateral grid to obtain the third bilateral grid; Tucker decomposition and Tucker reconstruction are respectively performed on the width of the second bilateral grid to obtain the fourth bilateral grid.

[0252] In another possible implementation, Tucker decomposition and Tucker reconstruction are respectively performed on the depth of the first bilateral grid to obtain the third bilateral grid; Tucker decomposition and Tucker reconstruction are respectively performed on the depth of the second bilateral grid to obtain the fourth bilateral grid.

[0253] 807. Upsampling processing is performed on the third bilateral grid to obtain the fifth bilateral grid, and upsampling processing is performed on the fourth bilateral grid to obtain the sixth bilateral grid.

[0254] For the third bilateral grid with texture information and the fourth bilateral grid with color information, upsampling processing can be respectively performed on the third bilateral grid and the fourth bilateral grid to correspondingly obtain the fifth bilateral grid with texture information and the sixth bilateral grid with color information. Specifically, the upsampling processing can be implemented in the following manner:

[0255] The second feature extraction layer is used to perform feature extraction processing on the image to be processed, obtaining a texture guidance map and a color guidance map. Then, the third bilateral grid is interpolated according to the texture guidance map to obtain a fifth bilateral grid, and the fourth bilateral grid is interpolated according to the color guidance map to obtain a sixth bilateral grid.

[0256] 808. An affine transformation is performed on the image to be processed according to the fifth bilateral grid to obtain a fifth feature, and an affine transformation is performed on the image to be processed according to the sixth bilateral grid to obtain a sixth feature.

[0257] After obtaining the fifth bilateral grid of texture information and the sixth bilateral grid of color information, the fifth bilateral grid can be used to perform an affine transformation on the image to be processed to obtain a fifth feature of the texture information of the image to be processed, and an affine transformation is performed on the image to be processed according to the sixth bilateral grid to obtain a sixth feature of the color information of the image to be processed.

[0258] 809. The fifth feature and the sixth feature are subjected to image reconstruction processing by the model to be trained to obtain a predicted image.

[0259] After obtaining the fifth feature of the texture information of the image to be processed and the sixth feature of the color information of the image to be processed, the fifth feature and the sixth feature can be input into the image reconstruction layer of the model to be trained, so that the image reconstruction layer performs image reconstruction processing on the fifth feature and the sixth feature to obtain a predicted image after image enhancement of the image to be processed. Specifically, the aforementioned image reconstruction processing can be implemented in the following manner:

[0260] The fifth feature and the sixth feature are subjected to a third fusion process by the image reconstruction layer to obtain a seventh feature. Then, the seventh feature is subjected to convolution processing by the image reconstruction layer to obtain a predicted image. The third fusion process includes at least one of normalization processing, summation processing, multiplication processing, concatenation processing, and concatenated convolution processing.

[0261] 810. According to the predicted image, the real image, and the processed predicted image, an objective loss is obtained. The objective loss is used to indicate the difference between the predicted image and the real image, and the difference between the predicted image and the processed predicted image.

[0262] After obtaining the predicted image, pixel flipping processing and blurring processing can be sequentially performed on the predicted image to obtain a processed predicted image. Then, calculations can be performed according to the predicted image, the real image, and the processed predicted image to obtain an objective loss. The objective loss can be used to indicate the difference between the predicted image and the real image, and the difference between the predicted image and the processed predicted image. Specifically, the objective loss can be calculated by the following formula:

[0263]

[0264]

[0265]

[0266] In the above formula, J j is the j-th image among the D training images (for example, the image to be processed), T j is the real image after image enhancement of J j , λ is a preset parameter, 255 is the maximum value of the pixels of the image to be processed, and mask is the prediction image obtained after performing pixel flipping processing on J j (for example, setting the value of each pixel in J j to the difference between the original value and 255, etc.), and then performing blurring processing (for example, performing Gaussian blurring processing with a certain radius, etc.).

[0267] 811. Update the model parameters of the model to be trained according to the target loss until the model training conditions are met, and obtain an image processing model.

[0268] After obtaining the target loss, the model parameters of the model to be trained can be updated based on the target loss, and the updated model to be trained can be trained using the next batch of training samples (that is, re-execute steps 801 to 811) until the model training conditions are met (for example, the target loss reaches convergence, etc.), and an image processing model can be obtained.

[0269] The image processing model obtained in the embodiments of the present application has the ability to perform image enhancement on the image to be processed. During the image enhancement process, the image processing model can incorporate the second feature of color information based on the first feature of texture information to obtain a third feature, and the image processing model can incorporate the first feature of texture information based on the second feature of color information to obtain a fourth feature. Therefore, the first bilateral grid generated based on the third feature not only contains the texture information of the image to be processed, but also contains the correlation between texture information and color information. The second bilateral grid generated based on the fourth feature not only contains the color information of the image to be processed, but also contains the correlation between texture information and color information. The information about the image to be processed contained in the two bilateral grids is relatively comprehensive. Then, the enhanced image to be processed obtained based on the first bilateral grid and the second bilateral grid can accurately display the image information that needs to be highlighted, thereby improving the image enhancement effect.

[0270] The above is a detailed description of the model training method provided in the embodiments of the present application. The image enhancement device and model training device provided in the embodiments of the present application will be introduced below. Figure 9 is a schematic structural diagram of an image enhancement device provided in an embodiment of the present application. AsFigure 9 As shown, the device includes:

[0271] A first acquisition module 901, configured to obtain a first feature of the texture information of the target image and a second feature of the color information of the target image through an image processing model;

[0272] A fusion module 902, configured to perform a first fusion process on the first feature and the second feature through an image processing model to obtain a third feature of the texture information, and perform a second fusion process on the first feature and the second feature through the image processing model to obtain a fourth feature of the color information;

[0273] A generation module 903, configured to generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature;

[0274] An enhancement module 904, configured to perform enhancement processing on the target image according to the first bilateral grid and the second bilateral network to obtain an enhanced target image.

[0275] In a possible implementation manner, the image processing model includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. The first acquisition module 901 is configured to: perform convolutional processing on the texture information of the target image through the N convolutional layers to obtain first sub-features respectively output by the N convolutional layers; perform convolutional processing on the color information of the target image through the N convolutional layers to obtain second sub-features respectively output by the N convolutional layers.

[0276] In a possible implementation manner, the image processing model further includes N first fusion layers and N second fusion layers. The fusion module 902 is configured to: perform convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; perform normalization processing on the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; perform convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1,..., N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

[0277] In a possible implementation, the device further includes: a second acquisition module, configured to acquire a target image; a downsampling module, configured to perform downsampling processing on the target image to obtain a downsampled target image; a separation module, configured to perform separation processing on the downsampled target image to obtain the texture information and color information of the target image.

[0278] In a possible implementation, the image processing model further includes an image reconstruction layer, and an enhancement module 904, configured to: perform mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and perform mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh, where the mesh reconstruction processing is used to remove some information in the bilateral mesh; perform upsampling processing on the third bilateral mesh to obtain a fifth bilateral mesh, and perform upsampling processing on the fourth bilateral mesh to obtain a sixth bilateral mesh; perform affine transformation on the target image according to the fifth bilateral mesh to obtain a fifth feature, and perform affine transformation on the target image according to the sixth bilateral mesh to obtain a sixth feature; perform image reconstruction processing on the fifth feature and the sixth feature through the image reconstruction layer to obtain an enhanced target image.

[0279] In a possible implementation, the enhancement module 904 is configured to: perform Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral mesh respectively to obtain a first bilateral mesh with processed height; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral mesh respectively to obtain a first bilateral mesh with processed width; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral mesh respectively to obtain a first bilateral mesh with processed depth; perform weighted summation processing on the first bilateral mesh with processed height, the first bilateral mesh with processed width, and the first bilateral mesh with processed depth to obtain a third bilateral mesh; perform Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral mesh respectively to obtain a second bilateral mesh with processed height; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral mesh respectively to obtain a second bilateral mesh with processed width; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral mesh respectively to obtain a second bilateral mesh with processed depth; perform weighted summation processing on the second bilateral mesh with processed height, the second bilateral mesh with processed width, and the second bilateral mesh with processed depth to obtain a fourth bilateral mesh.

[0280] In a possible implementation, the enhancement module 904 is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed; after obtaining the first bilateral grid with the height processed and the first bilateral grid with the width processed, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed; perform weighted summation processing on the second bilateral grid with the height processed and the second bilateral grid with the width processed to obtain a fourth bilateral grid.

[0281] In a possible implementation, the enhancement module 904 is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with the depth processed; after obtaining the first bilateral grid with the height processed and the first bilateral grid with the depth processed, weighted summation processing can be performed on these two bilateral grids to obtain a third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed; perform weighted summation processing on the second bilateral grid with the height processed and the second bilateral grid with the depth processed to obtain a fourth bilateral grid.

[0282] In a possible implementation, the enhancement module 904 is configured to: perform Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral grid respectively to obtain the first bilateral grid after width processing; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid respectively to obtain the first bilateral grid after depth processing; after obtaining the first bilateral grid after width processing and the first bilateral grid after depth processing, weighted summation processing can be performed on these two bilateral grids to obtain the third bilateral grid; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral grid respectively to obtain the second bilateral grid after width processing; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid respectively to obtain the second bilateral grid after depth processing; perform weighted summation processing on the second bilateral grid after width processing and the second bilateral grid after depth processing to obtain the fourth bilateral grid.

[0283] In a possible implementation, the enhancement module 904 is configured to: perform Tucker decomposition processing and Tucker reconstruction processing on the height of the first bilateral grid respectively to obtain the third bilateral grid; perform Tucker decomposition processing and Tucker reconstruction processing on the height of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0284] In a possible implementation, the enhancement module 904 is configured to: perform Tucker decomposition processing and Tucker reconstruction processing on the width of the first bilateral grid respectively to obtain the third bilateral grid; perform Tucker decomposition processing and Tucker reconstruction processing on the width of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0285] In a possible implementation, the enhancement module 904 is configured to: perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the first bilateral grid respectively to obtain the third bilateral grid; perform Tucker decomposition processing and Tucker reconstruction processing on the depth of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0286] In a possible implementation, the image processing model further includes a feature extraction layer, and the enhancement module 904 is configured to: perform feature extraction processing on the target image through the feature extraction layer to obtain a texture guidance map and a color guidance map; perform interpolation processing on the third bilateral grid according to the texture guidance map to obtain the fifth bilateral grid, and perform interpolation processing on the fourth bilateral grid according to the color guidance map to obtain the sixth bilateral grid.

[0287] In a possible implementation, the enhancement module 904 is configured to: perform a third fusion process on the fifth feature and the sixth feature through an image reconstruction layer to obtain a seventh feature; perform a convolution process on the seventh feature through the image reconstruction layer to obtain an enhanced target image.

[0288] In a possible implementation, the third fusion process includes at least one of a normalization process, a summation process, a multiplication process, a concatenation process, and a concatenated convolution process.

[0289] Figure 10 This is a schematic structural diagram of the model training device provided by the embodiments of the present application. As Figure 10 shown, the device includes:

[0290] A first acquisition module 1001, configured to acquire an image to be processed and a real image obtained by performing image enhancement on the image to be processed;

[0291] A second acquisition module 1002, configured to obtain a first feature of the texture information of the image to be processed and a second feature of the color information of the image to be processed through a model to be trained;

[0292] A fusion module 1003, configured to perform a first fusion process on the first feature and the second feature through the model to be trained to obtain a third feature of the texture information, and perform a second fusion process on the first feature and the second feature through the model to be trained to obtain a fourth feature of the color information;

[0293] A generation module 1004, configured to generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature;

[0294] An enhancement module 1005, configured to perform enhancement processing on the image to be processed according to the first bilateral grid and the second bilateral network to obtain a predicted image;

[0295] A third acquisition module 1006, configured to obtain a target loss according to the predicted image, the real image, and the processed predicted image, where the target loss is used to indicate the difference between the predicted image and the real image, and the difference between the predicted image and the processed predicted image;

[0296] An update module 1007, configured to update the model parameters of the model to be trained according to the target loss until the model training condition is satisfied to obtain an image processing model.

[0297] In a possible implementation, the processed predicted image is the predicted image after pixel flipping processing and blurring processing.

[0298] In a possible implementation, the model to be trained includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. The second obtaining module 1002 is configured to: perform convolutional processing on the texture information of the image to be processed through the N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; perform convolutional processing on the color information of the image to be processed through the N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

[0299] In a possible implementation, the model to be trained further includes N first fusion layers and N second fusion layers. The fusion module 1003 is configured to: perform convolutional processing and normalization processing on the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; perform normalization processing on the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; perform convolutional processing and addition processing on the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1, …, N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

[0300] In a possible implementation, the apparatus further includes: a fourth obtaining module, configured to obtain the image to be processed; a downsampling module, configured to perform downsampling processing on the image to be processed to obtain the downsampled image to be processed; a separation module, configured to perform separation processing on the downsampled image to be processed to obtain the texture information of the image to be processed and the color information of the image to be processed.

[0301] In a possible implementation, the model to be trained further includes an image reconstruction layer. The enhancement module 1005 is configured to: perform mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and perform mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh, where the mesh reconstruction processing is used to remove some information in the bilateral mesh; perform upsampling processing on the third bilateral mesh to obtain a fifth bilateral mesh, and perform upsampling processing on the fourth bilateral mesh to obtain a sixth bilateral mesh; perform affine transformation on the image to be processed according to the fifth bilateral mesh to obtain a fifth feature, and perform affine transformation on the image to be processed according to the sixth bilateral mesh to obtain a sixth feature; perform image reconstruction processing on the fifth feature and the sixth feature through the image reconstruction layer to obtain a predicted image.

[0302] In a possible implementation, the enhancement module 1005 is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with the depth processed; perform weighted summation on the first bilateral grid with the height processed, the first bilateral grid with the width processed, and the first bilateral grid with the depth processed to obtain the third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed; perform weighted summation on the second bilateral grid with the height processed, the second bilateral grid with the width processed, and the second bilateral grid with the depth processed to obtain the fourth bilateral grid.

[0303] In a possible implementation, the enhancement module 1005 is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed; after obtaining the first bilateral grid with the height processed and the first bilateral grid with the width processed, weighted summation can be performed on these two bilateral grids to obtain the third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed; perform weighted summation on the second bilateral grid with the height processed and the second bilateral grid with the width processed to obtain the fourth bilateral grid.

[0304] In a possible implementation, the enhancement module 1005 is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the first bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with the depth processed; after obtaining the first bilateral grid with the height processed and the first bilateral grid with the depth processed, weighted summation processing can be performed on these two bilateral grids to obtain the third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the second bilateral grid with the height processed; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed; perform weighted summation processing on the second bilateral grid with the height processed and the second bilateral grid with the depth processed to obtain the fourth bilateral grid.

[0305] In a possible implementation, the enhancement module 1005 is configured to: perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the first bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid respectively to obtain the first bilateral grid with the depth processed; after obtaining the first bilateral grid with the width processed and the first bilateral grid with the depth processed, weighted summation processing can be performed on these two bilateral grids to obtain the third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the second bilateral grid with the width processed; perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid respectively to obtain the second bilateral grid with the depth processed; perform weighted summation processing on the second bilateral grid with the width processed and the second bilateral grid with the depth processed to obtain the fourth bilateral grid.

[0306] In a possible implementation, the enhancement module 1005 is configured to: perform Tucker decomposition and Tucker reconstruction on the height of the first bilateral grid respectively to obtain the third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the height of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0307] In a possible implementation, the enhancement module 1005 is configured to: perform Tucker decomposition and Tucker reconstruction on the width of the first bilateral grid respectively to obtain the third bilateral grid; perform Tucker decomposition and Tucker reconstruction on the width of the second bilateral grid respectively to obtain the fourth bilateral grid.

[0308] In one possible implementation, the enhancement module 1005 is configured to: perform Tucker decomposition and Tucker reconstruction on the depth of the first bilateral grid to obtain a third bilateral grid; and perform Tucker decomposition and Tucker reconstruction on the depth of the second bilateral grid to obtain a fourth bilateral grid.

[0309] In one possible implementation, the model to be trained also includes a feature extraction layer and an enhancement module 1005, which is used to: perform feature extraction processing on the image to be processed through the feature extraction layer to obtain a texture guidance map and a color guidance map; interpolate the third bilateral grid according to the texture guidance map to obtain a fifth bilateral grid, and interpolate the fourth bilateral grid according to the color guidance map to obtain a sixth bilateral grid.

[0310] In a possible implementation, the enhancement module 1005 is configured to: perform a third fusion process on the fifth feature and the sixth feature through an image reconstruction layer to obtain a seventh feature; and perform a convolution process on the seventh feature through an image reconstruction layer to obtain a predicted image.

[0311] In a possible implementation, the third fusion process includes at least one of normalization processing, summation processing, multiplication processing, cascade processing, and cascade convolution processing.

[0312] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.

[0313] The embodiment of the present application also relates to an execution device, Figure 11 A structural diagram of the execution device provided in the embodiment of the present application. Figure 11 As shown, the execution device 1100 can be specifically manifested as a mobile phone, tablet, laptop, smart wearable device, server, etc., which is not limited here. Among them, the execution device 1100 can be deployed with Figure 9 The image enhancement device described in the corresponding embodiment is used to implement Figure 4 Specifically, the execution device 1100 includes: a receiver 1101, a transmitter 1102, a processor 1103 and a memory 1104 (wherein the number of the processor 1103 in the execution device 1100 can be one or more, Figure 10Take a processor as an example. Among them, the processor 1103 may include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103, and the memory 1104 may be connected by a bus or other means.

[0314] The memory 1104 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1103. A part of the memory 1104 may also include a non-volatile random access memory (NVRAM). The memory 1104 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, where the operation instructions may include various operation instructions for implementing various operations.

[0315] The processor 1103 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, where the bus system may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, all kinds of buses are referred to as the bus system in the figure.

[0316] The method disclosed in the embodiments of the present application can be applied to or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor 1103 or the instructions in the form of software. The above-mentioned processor 1103 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1103 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 1104, and the processor 1103 reads the information in the memory 1104 and combines its hardware to complete the steps of the above method.

[0317] The receiver 1101 can be used to receive input digital or character information, and generate signal inputs related to the relevant settings and function controls of the execution device. The transmitter 1102 can be used to output digital or character information through the first interface; the transmitter 1102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1102 can also include a display device such as a display screen.

[0318] In the embodiments of the present application, in one case, the processor 1103 is used to Figure 4 perform image enhancement on the image through the

[0319] The embodiments of the present application also relate to a training device, Figure 12 which is a schematic structural diagram of the training device provided by the embodiments of the present application. As Figure 12As shown, the training device 1200 is implemented by one or more servers. The training device 1200 can vary significantly due to configuration or performance differences, and may include one or more central processing units (CPUs) 1214 (e.g., one or more processors) and a memory 1232, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1242 or data 1244. Among them, the memory 1232 and the storage media 1230 can be transient storage or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the training device. Further, the central processing unit 1214 can be configured to communicate with the storage media 1230 and execute a series of instruction operations in the storage media 1230 on the training device 1200.

[0320] The training device 1200 may further include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258; or, one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so on.

[0321] Specifically, the training device can execute Figure 8 the model training method in the corresponding embodiment.

[0322] The embodiment of the present application also relates to a computer storage medium. The computer-readable storage medium stores a program for signal processing. When it runs on a computer, it causes the computer to execute the steps performed by the foregoing execution device, or causes the computer to execute the steps performed by the foregoing training device.

[0323] The embodiment of the present application also relates to a computer program product. The computer program product stores instructions. When the instructions are executed by a computer, they cause the computer to execute the steps performed by the foregoing execution device, or cause the computer to execute the steps performed by the foregoing training device.

[0324] The execution device, training device, or terminal device provided by the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be a processor, for example, and the communication unit may be an input / output interface, a pin, a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the data processing method described in the above embodiments, or to cause the chip in the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit inside the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit outside the chip in the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0325] Specifically, please refer to Figure 13 , Figure 13 which is a schematic structural diagram of the chip provided by the embodiments of the present application. The chip may be embodied as a neural network processor NPU 1300. The NPU 1300 is mounted on the main CPU (Host CPU) as a coprocessor, and tasks are assigned by the Host CPU. The core part of the NPU is the arithmetic circuit 1303. The arithmetic circuit 1303 is controlled by the controller 1304 to extract matrix data from the memory and perform multiplication operations.

[0326] In some implementations, the arithmetic circuit 1303 includes multiple processing units (Process Engine, PE) inside. In some implementations, the arithmetic circuit 1303 is a two-dimensional systolic array. The arithmetic circuit 1303 may also be a one-dimensional systolic array or other electronic circuits that can perform mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1303 is a general matrix processor.

[0327] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 1302 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 1301 and performs matrix operations with matrix B, and the partial results or final results of the obtained matrix are saved in the accumulator 1308.

[0328] The unified memory 1306 is used to store input data and output data. The weight data directly passes through the Direct Memory Access Controller (DMAC) 1305 and is transferred to the weight memory 1302 by the DMAC. The input data is also transferred to the unified memory 1306 by the DMAC.

[0329] The BIU is the Bus Interface Unit, that is, the bus interface unit 1313, which is used for the interaction between the AXI bus, the DMAC, and the Instruction Fetch Buffer (IFB) 1309.

[0330] The bus interface unit 1313 (Bus Interface Unit, abbreviated as BIU) is used for the instruction fetch buffer 1309 to obtain instructions from the external memory, and is also used for the storage unit access controller 1305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0331] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1306, or transfer the weight data to the weight memory 1302, or transfer the input data to the input memory 1301.

[0332] The vector calculation unit 1307 includes multiple arithmetic processing units, which, if necessary, further process the output of the arithmetic circuit 1303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolution / full connection layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of the prediction label plane, etc.

[0333] In some implementations, the vector calculation unit 1307 can store the processed output vector in the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function; or, a non-linear function to the output of the arithmetic circuit 1303, such as performing linear interpolation on the prediction label plane extracted by the convolutional layer, or, for another example, a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit 1307 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as the activation input to the arithmetic circuit 1303, for example, for use in subsequent layers in a neural network.

[0334] The instruction fetch buffer 1309 connected to the controller 1304 is used to store the instructions used by the controller 1304;

[0335] The unified memory 1306, the input memory 1301, the weight memory 1302, and the instruction fetch memory 1309 are all On-Chip memories. The external memory is private to the NPU hardware architecture.

[0336] Among them, the processor mentioned anywhere above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above programs.

[0337] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between the modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

[0338] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by means of dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for this application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disc of a computer, and includes several instructions for causing a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in various embodiments of this application.

[0339] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0340] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (such as coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. An image enhancement method, characterized in that, The method includes: Obtaining a first feature of the texture information of the target image and a second feature of the color information of the target image through an image processing model; Performing a first fusion process on the first feature and the second feature through the image processing model to obtain a third feature of the texture information, and performing a second fusion process on the first feature and the second feature through the image processing model to obtain a fourth feature of the color information; Generating a first bilateral grid according to the third feature and generating a second bilateral grid according to the fourth feature; Performing enhancement processing on the target image according to the first bilateral grid and the second bilateral network to obtain an enhanced target image.

2. The method according to claim 1, wherein The image processing model includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. The obtaining of the first feature of the texture information of the target image and the second feature of the color information of the target image through the image processing model includes: Performing convolutional processing on the texture information of the target image through the N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; Performing convolutional processing on the color information of the target image through the N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

3. The method according to claim 2, wherein The image processing model further includes N first fusion layers and N second fusion layers. The performing of the first fusion process on the first feature and the second feature through the image processing model to obtain the third feature of the texture information, and the performing of the second fusion process on the first feature and the second feature through the image processing model to obtain the fourth feature of the color information includes: Processing the (i-1)-th third sub-feature and the (i-1)-th fourth sub-feature output by the (i-1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; Processing the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; Processing the (i-1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1,..., N; when i = 1, the (i-1)-th third sub-feature is the first first sub-feature, and the (i-1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

4. The method according to any one of claims 1 to 3, characterized in that Before the obtaining of the first feature of the texture information of the target image and the second feature of the color information of the target image through the image processing model, the method further includes: Obtaining a target image; Performing downsampling processing on the target image to obtain a downsampled target image; Performing separation processing on the downsampled target image to obtain the texture information of the target image and the color information of the target image.

5. The method according to any one of claims 1 to 3, characterized in that, The image processing model further includes an image reconstruction layer. The performing of the enhancement processing on the target image according to the first bilateral grid and the second bilateral network to obtain an enhanced target image includes: Perform mesh reconstruction processing on the first bilateral mesh to obtain a third bilateral mesh, and perform mesh reconstruction processing on the second bilateral mesh to obtain a fourth bilateral mesh. The mesh reconstruction processing is used to remove some information in the bilateral mesh; Perform upsampling processing on the third bilateral mesh to obtain a fifth bilateral mesh, and perform upsampling processing on the fourth bilateral mesh to obtain a sixth bilateral mesh; Perform affine transformation on the target image according to the fifth bilateral mesh to obtain a fifth feature, and perform affine transformation on the target image according to the sixth bilateral mesh to obtain a sixth feature; Perform image reconstruction processing on the fifth feature and the sixth feature through an image reconstruction layer to obtain an enhanced target image.

6. A model training method, characterized in that, The method includes: Obtain an image to be processed and a real image after image enhancement of the image to be processed; Obtain a first feature of the texture information of the image to be processed and a second feature of the color information of the image to be processed through a model to be trained; Perform a first fusion process on the first feature and the second feature through the model to be trained to obtain a third feature of the texture information, and perform a second fusion process on the first feature and the second feature through the model to be trained to obtain a fourth feature of the color information; Generate a first bilateral mesh according to the third feature, and generate a second bilateral mesh according to the fourth feature; Perform enhancement processing on the image to be processed according to the first bilateral mesh and the second bilateral network to obtain a predicted image; Obtain a target loss according to the predicted image, the real image, and the processed predicted image. The target loss is used to indicate the difference between the predicted image and the real image, and the difference between the predicted image and the processed predicted image; Update the model parameters of the model to be trained according to the target loss until the model training condition is satisfied to obtain an image processing model.

7. The method according to claim 6, characterized in that, The processed predicted image is the predicted image after pixel flipping processing and blur processing.

8. The method according to claim 6 or 7, characterized in that The model to be trained includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. Obtaining a first feature of the texture information of the image to be processed and a second feature of the color information of the image to be processed through the model to be trained includes: Perform convolution processing on the texture information of the image to be processed through the N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; Perform convolution processing on the color information of the image to be processed through the N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

9. The method according to claim 8, wherein The model to be trained further includes N first fusion layers and N second fusion layers. Performing a first fusion process on the first feature and the second feature through the model to be trained to obtain a third feature of the texture information, and performing a second fusion process on the first feature and the second feature through the model to be trained to obtain a fourth feature of the color information includes: The i-th third sub-feature is obtained by processing the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer; The i-th fifth sub-feature is obtained by processing the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer; The i-th fourth sub-feature is obtained by processing the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer; where i = 1, …, N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

10. The method according to claim 6 or 7, characterized in that Before obtaining the first feature of the texture information of the image to be processed and the second feature of the color information of the image to be processed through the model to be trained, the method further includes: Obtaining an image to be processed; Performing downsampling processing on the image to be processed to obtain a downsampled image to be processed; Performing separation processing on the downsampled image to be processed to obtain the texture information of the image to be processed and the color information of the image to be processed.

11. The method according to claim 6 or 7, characterized in that The model to be trained further includes an image reconstruction layer. The step of obtaining a predicted image by enhancing the image to be processed according to the first bilateral grid and the second bilateral network includes: Performing grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and performing grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid. The grid reconstruction processing is used to remove some information in the bilateral grid; Performing upsampling processing on the third bilateral grid to obtain a fifth bilateral grid, and performing upsampling processing on the fourth bilateral grid to obtain a sixth bilateral grid; Performing an affine transformation on the image to be processed according to the fifth bilateral grid to obtain a fifth feature, and performing an affine transformation on the image to be processed according to the sixth bilateral grid to obtain a sixth feature; Performing image reconstruction processing on the fifth feature and the sixth feature through the image reconstruction layer to obtain a predicted image.

12. An image enhancement device, characterized in that, The device includes: A first acquisition module, configured to obtain a first feature of the texture information of a target image and a second feature of the color information of the target image through an image processing model; A fusion module, configured to perform first fusion processing on the first feature and the second feature through the image processing model to obtain a third feature of the texture information, and perform second fusion processing on the first feature and the second feature through the image processing model to obtain a fourth feature of the color information; A generation module, configured to generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature; An enhancement module, configured to perform enhancement processing on the target image according to the first bilateral grid and the second bilateral network to obtain an enhanced target image.

13. The device according to claim 12, wherein The image processing model includes N convolutional layers connected in sequence. The first feature includes N first sub-features, the second feature includes N second sub-features, and the first acquisition module is configured to: Performing convolution processing on the texture information of the target image through the N convolutional layers to obtain first sub-features respectively output by the N convolutional layers; Performing convolution processing on the color information of the target image through the N convolutional layers to obtain second sub-features respectively output by the N convolutional layers.

14. The device according to claim 13, characterized in that, The image processing model further includes N first fusion layers and N second fusion layers. The fusion module is configured to: Process the (i - 1)-th third sub-feature and the (i - 1)-th fourth sub-feature output by the (i - 1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; Process the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; Process the (i - 1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1, …, N; when i = 1, the (i - 1)-th third sub-feature is the first first sub-feature, and the (i - 1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

15. The device according to any one of claims 12 to 14, characterized in that The device further includes: A second acquisition module, configured to acquire a target image; A downsampling module, configured to perform downsampling processing on the target image to obtain a downsampled target image; A separation module, configured to perform separation processing on the downsampled target image to obtain the texture information of the target image and the color information of the target image.

16. The device according to any one of claims 12 to 14, characterized in that, The enhancement module is configured to: Perform grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and perform grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid. The grid reconstruction processing is used to remove partial information in the bilateral grid; Perform upsampling processing on the third bilateral grid to obtain a fifth bilateral grid, and perform upsampling processing on the fourth bilateral grid to obtain a sixth bilateral grid; Perform affine transformation on the target image according to the fifth bilateral grid to obtain a fifth feature, and perform affine transformation on the target image according to the sixth bilateral grid to obtain a sixth feature; Perform image reconstruction processing on the fifth feature and the sixth feature through an image reconstruction layer to obtain an enhanced target image.

17. A model training device, characterized in that, The device includes: A first acquisition module, configured to acquire an image to be processed and a real image after image enhancement of the image to be processed; A second acquisition module, configured to obtain a first feature of the texture information of the image to be processed and a second feature of the color information of the image to be processed through a model to be trained; A fusion module, configured to perform first fusion processing on the first feature and the second feature through the model to be trained to obtain a third feature of the texture information, and perform second fusion processing on the first feature and the second feature through the model to be trained to obtain a fourth feature of the color information; A generation module, configured to generate a first bilateral grid according to the third feature and generate a second bilateral grid according to the fourth feature; An enhancement module for enhancing the image to be processed according to the first bilateral grid and the second bilateral network to obtain a predicted image; A third acquisition module for obtaining a target loss according to the predicted image, the real image, and the processed predicted image, where the target loss is used to indicate the difference between the predicted image and the real image, and the difference between the predicted image and the processed predicted image; An update module for updating the model parameters of the model to be trained according to the target loss until the model training condition is satisfied to obtain an image processing model.

18. The device according to claim 17, characterized in that, The processed predicted image is the predicted image after pixel flipping processing and blurring processing.

19. The device according to claim 17 or 18, characterized in that The model to be trained includes N convolutional layers connected in sequence. The first feature includes N first sub-features, and the second feature includes N second sub-features. The second acquisition module is used for: Performing convolutional processing on the texture information of the image to be processed through the N convolutional layers to obtain the first sub-features respectively output by the N convolutional layers; Performing convolutional processing on the color information of the image to be processed through the N convolutional layers to obtain the second sub-features respectively output by the N convolutional layers.

20. The device according to claim 19, characterized in that, The model to be trained further includes N first fusion layers and N second fusion layers. The fusion module is used for: Processing the (i-1)-th third sub-feature and the (i-1)-th fourth sub-feature output by the (i-1)-th first fusion layer through the i-th first fusion layer to obtain the i-th third sub-feature; Processing the i-th first sub-feature and the i-th second sub-feature through the i-th second fusion layer to obtain the i-th fifth sub-feature; Processing the (i-1)-th fourth sub-feature and the i-th fifth sub-feature through the i-th first fusion layer to obtain the i-th fourth sub-feature; where i = 1,..., N; when i = 1, the (i-1)-th third sub-feature is the first first sub-feature, and the (i-1)-th fourth sub-feature is the first second sub-feature; when i = N, the i-th third sub-feature is the third feature, and the i-th fourth sub-feature is the fourth feature.

21. The device according to claim 17 or 18, characterized in that, The apparatus further includes: A fourth acquisition module for acquiring an image to be processed; A downsampling module for performing downsampling processing on the image to be processed to obtain the downsampled image to be processed; A separation module for performing separation processing on the downsampled image to be processed to obtain the texture information of the image to be processed and the color information of the image to be processed.

22. The device according to claim 17 or 18, characterized in that, The model to be trained further includes an image reconstruction layer. The enhancement module is used for: Performing grid reconstruction processing on the first bilateral grid to obtain a third bilateral grid, and performing grid reconstruction processing on the second bilateral grid to obtain a fourth bilateral grid. The grid reconstruction processing is used to remove some information in the bilateral grid; Performing upsampling processing on the third bilateral grid to obtain a fifth bilateral grid, and performing upsampling processing on the fourth bilateral grid to obtain a sixth bilateral grid; Perform an affine transformation on the image to be processed according to the fifth bilateral grid to obtain a fifth feature, and perform an affine transformation on the image to be processed according to the sixth bilateral grid to obtain a sixth feature; Perform image reconstruction processing on the fifth feature and the sixth feature through an image reconstruction layer to obtain a predicted image.

23. An image enhancement device, characterized in that, The apparatus includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the image processing apparatus executes the method according to any one of claims 1 to 11.

24. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, and when the instructions are executed by one or more computers, the one or more computers implement the method according to any one of claims 1 to 11.

25. A computer program product, characterized in that, The computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the method according to any one of claims 1 to 11.

Images