Image enhancement method, device and robot based on gray world hypothesis theory

Abstract

The present invention provides an image enhancement method, device and robot based on the grayscale world hypothesis theory, which relates to the field of computer vision technology. In this method, the color information and structural information of the original image are decoupled, and the structural information is restored by a grayscale transformation network, and the color information is restored by a re-rendering network, which can improve the naturalness and color consistency of the image enhancement result. The re-rendering network used in this method includes a first feature extraction branch, a second feature extraction branch, a dual-feature fusion module and a decoding layer. The first feature of the original image is fused with the second feature of the grayscale image through the dual-feature fusion module, which can make full use of the color information of the original image, so that the final image enhancement result has better color consistency and more accurate exposure rate, and is more natural and colorful.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image enhancement method, device and robot based on grayscale world hypothesis theory. Background Art

[0002] In recent years, with the continuous development of image processing, using images to obtain information has become an important means. For example, high-quality images are used for target detection. However, due to the inevitable limitations of the lighting environment and imaging technology during image acquisition, such as backlight, weak light, uneven lighting, and insufficient exposure time, this will result in a low image contribution rate, making the acquired image blurry, low contrast, and lack of details.

[0003] Existing low-brightness image enhancement algorithms usually achieve enhancement by simply constraining image gradients. The enhancement results are often accompanied by distortion and detail loss, and are unable to effectively extract useful image information. Summary of the invention

[0004] The present invention provides an image enhancement method, device and robot based on gray world hypothesis theory, so as to solve the defects existing in the prior art.

[0005] The present invention provides an image enhancement method based on grayscale world hypothesis theory, comprising:

[0006] Based on the grayscale world hypothesis theory, the grayscale transformation network is applied to predict the grayscale image of the original image;

[0007] Based on the re-rendering network, applying the original image to re-render the grayscale image, restoring the color of the grayscale image, and obtaining an image enhancement result;

[0008] The re-rendering network includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer, the first feature extraction branch is used to extract the first feature of the original image, the second feature extraction branch is used to extract the second feature of the grayscale image, the dual feature fusion module is used to fuse the first feature with the second feature to obtain a fusion result, and the decoding layer is used to obtain the image enhancement result based on the fusion result;

[0009] The dual-feature fusion module includes an input layer, a first depth-separable convolutional layer, a multi-head attention layer, a second depth-separable convolutional layer and a fusion output layer connected in sequence, and the input layer is connected to the fusion output layer;

[0010] The input layer is used to receive input features;

[0011] The first depth-wise separable convolutional layer is used to perform a convolution operation on the input feature to obtain a first output result;

[0012] The multi-head attention layer is used to extract attention features based on the first output result;

[0013] The second depth-wise separable convolutional layer is used to perform a convolution operation on the attention feature to obtain a second output result;

[0014] The fusion output layer is used to fuse the second output result with the input feature to obtain an output feature.

[0015] According to an image enhancement method based on grayscale world hypothesis theory provided by the present invention, the multi-head attention layer includes a flattening layer, a matrix multiplication layer and a color mapping layer;

[0016] The flattening layer is used to flatten the first output result into a one-dimensional tensor, where the one-dimensional tensor includes a query vector, a key vector, and a value vector;

[0017] The matrix multiplication layer is used to perform matrix multiplication on the query vector and the key vector to obtain a transformation matrix;

[0018] The color mapping layer is used to transform the value vector based on the transformation matrix to obtain the attention feature.

[0019] According to an image enhancement method based on the grayscale world hypothesis theory provided by the present invention, the convolution operations of the first depth-separable convolution layer and the second depth-separable convolution layer both include partial convolution operations and depth-separable convolution operations.

[0020] According to an image enhancement method based on grayscale world hypothesis theory provided by the present invention, the first feature extraction branch and the second feature extraction branch both include feature extraction layers of multiple scales, and each feature extraction layer of each scale is correspondingly connected to one of the dual feature fusion modules;

[0021] A plurality of the dual-feature fusion modules are all connected to the decoding layer.

[0022] According to an image enhancement method based on the grayscale world hypothesis theory provided by the present invention, the grayscale transformation network and the re-rendering network are jointly trained based on the following steps:

[0023] Inputting the low brightness image sample into the initial grayscale transformation network to obtain the grayscale image sample output by the initial grayscale transformation network;

[0024] Inputting the low brightness image sample and the grayscale image sample into an initial re-rendering network to obtain an enhanced image sample output by the initial re-rendering network;

[0025] Calculating a first L1 loss based on the grayscale image samples and the normal brightness grayscale image samples corresponding to the low brightness image samples, and calculating a second L1 loss and a structural similarity loss based on the enhanced image samples and the normal brightness image samples corresponding to the low brightness image samples;

[0026] Based on the first L1 loss, the second L1 loss and the structural similarity loss, structural parameters of the initial grayscale transformation network and the initial re-rendering network are iteratively updated to obtain the grayscale transformation network and the re-rendering network.

[0027] The present invention also provides an image enhancement device, comprising:

[0028] Grayscale image prediction module, used to predict the grayscale image of the original image based on the grayscale world hypothesis theory and the grayscale transformation network;

[0029] An image enhancement module, used to apply the original image to re-render the grayscale image based on a re-rendering network, restore the color of the grayscale image, and obtain an image enhancement result;

[0030] The re-rendering network includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer, the first feature extraction branch is used to extract the first feature of the original image, the second feature extraction branch is used to extract the second feature of the grayscale image, the dual feature fusion module is used to fuse the first feature with the second feature to obtain a fusion result, and the decoding layer is used to obtain the image enhancement result based on the fusion result;

[0031] The dual-feature fusion module includes an input layer, a first depth-separable convolutional layer, a multi-head attention layer, a second depth-separable convolutional layer and a fusion output layer connected in sequence, and the input layer is connected to the fusion output layer;

[0032] The input layer is used to receive input features;

[0033] The first depth-wise separable convolutional layer is used to perform a convolution operation on the input feature to obtain a first output result;

[0034] The multi-head attention layer is used to extract attention features based on the first output result;

[0035] The second depth-wise separable convolutional layer is used to perform a convolution operation on the attention feature to obtain a second output result;

[0036] The fusion output layer is used to fuse the second output result with the input feature to obtain an output feature.

[0037] The present invention also provides a robot, comprising a controller, a robot body, and an image acquisition device mounted on the robot body, wherein the controller is connected to the robot body and the image acquisition device;

[0038] The image acquisition device is used to acquire the original image of the robot body during the working process, and send the original image to the controller;

[0039] The controller is used to determine the image enhancement result of the original image based on the above-mentioned image enhancement method based on the gray world hypothesis theory, and control the robot body based on the image enhancement result.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image enhancement method based on the grayscale world hypothesis theory as described above is implemented.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described image enhancement methods based on the grayscale world hypothesis theory.

[0042] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image enhancement method based on the gray world hypothesis theory as described above is implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The image enhancement method, device and robot based on the grayscale world hypothesis theory provided by the present invention firstly apply the grayscale transformation network based on the grayscale world hypothesis theory to predict the grayscale image of the original image; then based on the re-rendering network, apply the original image to re-render the grayscale image, restore the color of the grayscale image, and obtain the image enhancement result. In this method, the color information and structural information of the original image are decoupled, and the structural information is restored by using the grayscale transformation network, and the color information is restored by using the re-rendering network, which can improve the naturalness and color consistency of the image enhancement result. The re-rendering network used in this method includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer. The first feature of the original image is fused with the second feature of the grayscale image through the dual feature fusion module, which can make full use of the color information of the original image, so that the final image enhancement result has better color consistency and more accurate exposure rate, and is more natural and colorful. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without paying any creative work.

[0046] Figure 1 It is a flow chart of an image enhancement method based on the gray world hypothesis theory provided by the present invention;

[0047] Figure 2 It is a structural schematic diagram of a dual-feature fusion module in an image enhancement method based on the gray world hypothesis theory provided by the present invention;

[0048] Figure 3 It is a structural schematic diagram of a re-rendering network in an image enhancement method based on the gray world hypothesis theory provided by the present invention;

[0049] Figure 4 It is a schematic diagram of comparison between an original image and an image enhancement result in an image enhancement method based on the gray world hypothesis theory provided by the present invention;

[0050] Figure 5 is a schematic structural diagram of an image enhancement device provided by the present invention;

[0051] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] When the robot works outdoors, the images taken in low-brightness environments are severely degraded and visibility is low, resulting in reduced target recognition capabilities, which in turn seriously affects advanced visual tasks such as visual navigation, traffic monitoring, and remote sensing. Existing low-brightness image enhancement algorithms usually achieve enhancement by simply constraining image gradients. The enhancement results are often accompanied by distortion and loss of details, and are unable to effectively extract useful image information, resulting in the resulting images still having a low contribution rate and being unable to perform subsequent applications such as target detection. Based on this, an image enhancement method based on the grayscale world hypothesis theory is provided in an embodiment of the present invention.

[0054] Figure 1FIG. 1 is a flow chart of an image enhancement method based on the gray world hypothesis theory provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0055] S1, based on the gray world hypothesis theory, applies the gray transformation network to predict the gray image of the original image;

[0056] S2, based on the re-rendering network, applying the original image to re-render the grayscale image, restoring the color of the grayscale image, and obtaining an image enhancement result;

[0057] Among them, the re-rendering network includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer, the first feature extraction branch is used to extract the first feature of the original image, the second feature extraction branch is used to extract the second feature of the grayscale image, the dual feature fusion module is used to fuse the first feature with the second feature to obtain a fusion result, and the decoding layer is used to obtain the image enhancement result based on the fusion result.

[0058] Specifically, the image enhancement method based on the gray world hypothesis theory provided in the embodiment of the present invention is executed by an image enhancement device, which can be configured in a computer. The computer can be a local computer, an offline computer or a cloud computer. The local computer can be a computer, a tablet, etc., and the offline computer can be an industrial control board, etc., which is not specifically limited here.

[0059] First, step S1 is performed to predict the grayscale image of the original image by using the grayscale world hypothesis theory and applying the grayscale transformation network. The original image is a color image acquired by an image acquisition device and is a low-brightness image that needs to be enhanced in brightness. The image acquisition device can be a camera and can be mounted on a robot.

[0060] This step focuses on restoring the structure and texture of the original image without considering the color information, that is, the grayscale transformation network predicts the grayscale image after brightness enhancement. The grayscale transformation network can be a G-Net, which is similar to the U-Net in form and can include multiple encoders with gradually decreasing scales and multiple decoders with gradually increasing scales connected in sequence. The number of scales of the encoder is the same as the number of scales of the decoder, and the encoders and decoders of the same scale are connected accordingly. The encoder can include a convolution activation layer (Conv+Leaky Relu), and the decoder can include a transposed convolution layer (Transpose Conv) and a convolution activation layer (Conv+Relu). The encoder can be specifically connected to the convolution activation layer in the decoder of the same scale.

[0061] Here, the original image can be input into the grayscale transformation network, and the grayscale transformation network outputs a grayscale image of the original image. The grayscale image contains rich structure and texture information, but does not contain color information.

[0062] Then, step S2 is performed, using the re-rendering network, applying the original image to re-render the grayscale image, restoring the color of the grayscale image, and obtaining an image enhancement result. Here, the original image and the grayscale image can be input into the re-rendering network, and the re-rendering network outputs the image enhancement result of the original image.

[0063] The re-rendering network may be an R-Net, which may include a first feature extraction branch, a second feature extraction branch, a dual feature fusion module, and a decoding layer. The first feature extraction branch may be used to extract a first feature of the original image, the second feature extraction branch may be used to extract a second feature of the grayscale image, and the dual feature fusion module (Dual Feature Enhancement, DFE) module may be used to fuse the first feature with the second feature to obtain a fusion result.

[0064] The decoding layer is used to decode the fusion result to obtain the image enhancement result.

[0065] The image enhancement method based on the grayscale world hypothesis theory provided in the embodiment of the present invention first applies the grayscale transformation network based on the grayscale world hypothesis theory to predict the grayscale image of the original image; then, based on the re-rendering network, the original image is applied to re-render the grayscale image to restore the color of the grayscale image and obtain the image enhancement result. In this method, the color information and structural information of the original image are decoupled, and the structural information is restored by the grayscale transformation network, and the color information is restored by the re-rendering network, which can improve the naturalness and color consistency of the image enhancement result. The re-rendering network used in this method includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer. The first feature of the original image is fused with the second feature of the grayscale image through the dual feature fusion module, which can make full use of the color information of the original image, so that the final image enhancement result has better color consistency and more accurate exposure, and is more natural and colorful.

[0066] On the basis of the above embodiment, the dual-feature fusion module includes an input layer, a first depth-separable convolution layer, a multi-head attention layer, a second depth-separable convolution layer and a fusion output layer connected in sequence, and the input layer is connected to the fusion output layer;

[0067] The input layer is used to receive input features;

[0068] The first depth-wise separable convolutional layer is used to perform a convolution operation on the input feature to obtain a first output result;

[0069] The multi-head attention layer is used to extract attention features based on the first output result;

[0070] The second depth-wise separable convolutional layer is used to perform a convolution operation on the attention feature to obtain a second output result;

[0071] The fusion output layer is used to fuse the second output result with the input feature to obtain an output feature.

[0072] Specifically, Figure 2 As shown, the dual feature fusion module includes an input layer, a first depth-wise separable convolution (DConv) layer, a multi-head attention layer, a second depth-wise separable convolution layer, and a fusion output layer, which are connected in sequence, and the input layer is connected to the fusion output layer.

[0073] The input layer is used to receive input features, which can be the concatenation of the first feature and the second feature. The input feature can be expressed as , C is the number of input feature channels, H is the height of the input feature, and W is the width of the input feature.

[0074] The first depth-wise separable convolutional layer is used to perform a convolution operation on the input features obtained by the input layer to obtain a first output result. The first depth-wise separable convolutional layer can introduce a partial convolution (PConv) operation, point-by-point convolution, and first increase the number of channels, so that subsequent depth convolutions can operate in a higher-dimensional feature space, increasing the expressiveness of the re-rendering network. This convolutional layer can significantly reduce the amount of calculation and the amount of parameters. Therefore, the first depth-wise separable convolutional layer can be an inverse depth-wise separable convolutional layer, which can be expressed as , the first output result is a multidimensional tensor.

[0075] The multi-head attention layer is used to extract attention features based on the first output result. The multi-head attention layer includes a flattening layer, a matrix multiplication layer, and a color mapping layer. The flattening layer is used to flatten the first output result into a one-dimensional tensor to obtain a query vector Q, a key vector K, and a value vector . That is:

[0076] ;

[0077] in, Flattened layer.

[0078] The query vector Q and the key vector K are input into the matrix multiplication layer, and the query vector Q and the key vector K are matrix multiplied by the matrix multiplication layer to obtain the transformation matrix .

[0079] That is: ;

[0080] in, is the normalization layer, is the normalization coefficient.

[0081] The transformation matrix and the value vector V are input into the color mapping layer, and the color mapping layer transforms the value vector V using the transformation matrix M, performs feature-level color mapping, and obtains the attention feature.

[0082] The second depth-wise separable convolutional layer is used to perform a convolution operation on the attention feature to obtain a second output result.

[0083] The fusion output layer is used to fuse the second output result with the input feature to obtain the output feature.

[0084] In the embodiment of the present invention, a matrix color correction (MCC) block can be formed by the first depth-separable convolution layer, the multi-head attention layer, and the second depth-separable convolution layer in the dual-feature fusion module, and convolution color enhancement and local refinement can be performed. The global receptive field and channel operation of the dual-feature fusion module well implement color correction and improve the performance of the re-rendering network.

[0085] On the basis of the above embodiment, the first feature extraction branch and the second feature extraction branch each include feature extraction layers of multiple scales, and each feature extraction layer of each scale is correspondingly connected to one of the dual feature fusion modules;

[0086] A plurality of the dual-feature fusion modules are all connected to the decoding layer.

[0087] Specifically, Figure 3 As shown, the first feature extraction branch and the second feature extraction branch may include feature extraction layers of 5 scales, and each scale of the feature extraction layer is correspondingly connected to a dual feature fusion module.

[0088] The feature extraction layer of each scale in the first feature extraction branch is used to extract the first feature of a granularity, and the feature extraction layer of each scale in the second feature extraction branch is used to extract the second feature of a granularity. The dual feature fusion module connected to the feature extraction layer of each scale is used to fuse the first feature and the second feature of the granularity corresponding to the scale to obtain the fusion result of the granularity.

[0089] Each dual-feature fusion module is connected to the decoding layer, and the decoding layer obtains the final image enhancement result through the fusion results obtained by each dual-feature fusion module. Here, the decoding layer may include a concatenated convolution layer (Concat+Conv).

[0090] In the embodiment of the present invention, by setting feature extraction layers of multiple scales, first features and second features of different granularities can be extracted, thereby making the obtained image enhancement result more accurate.

[0091] Based on the above embodiment, the convolution operations of the first depthwise separable convolution layer and the second depthwise separable convolution layer both include partial convolution operations and depthwise separable convolution operations. Through partial convolution operations, redundant calculations and memory accesses can be reduced, and the floating-point operation speed can be improved, thereby solving the problem of low floating-point operation speed in depthwise separable convolution operations and improving the calculation speed and efficiency of the re-rendering network.

[0092] Based on the above embodiment, the grayscale transformation network and the re-rendering network are jointly trained based on the following steps:

[0093] Inputting the low brightness image sample into the initial grayscale transformation network to obtain the grayscale image sample output by the initial grayscale transformation network;

[0094] Inputting the low brightness image sample and the grayscale image sample into an initial re-rendering network to obtain an enhanced image sample output by the initial re-rendering network;

[0095] Calculating a first L1 loss based on the grayscale image samples and the normal brightness grayscale image samples corresponding to the low brightness image samples, and calculating a second L1 loss and a structural similarity loss based on the enhanced image samples and the normal brightness image samples corresponding to the low brightness image samples;

[0096] Based on the first L1 loss, the second L1 loss and the structural similarity loss, structural parameters of the initial grayscale transformation network and the initial re-rendering network are iteratively updated to obtain the grayscale transformation network and the re-rendering network.

[0097] Specifically, in the process of training the initial grayscale transformation network to obtain the grayscale transformation network and training the initial re-rendering network to obtain the re-rendering network, the low-brightness image sample can be first input into the initial grayscale transformation network to obtain the grayscale image sample output by the initial grayscale transformation network. The processing process of the initial grayscale transformation network can be expressed as: ;

[0098] in, is a grayscale image sample, is a low brightness image sample, is the function representation of the initial grayscale transformation network.

[0099] After that, the low brightness image samples and grayscale image samples are input into the initial re-rendering network to obtain the enhanced image samples output by the initial re-rendering network. The processing process of the initial re-rendering network can be expressed as:

[0100] ;

[0101] in, To enhance the image samples, is the function representation of the initial re-rendering network.

[0102] Afterwards, using the grayscale image sample With low brightness image sample Corresponding normal brightness grayscale image samples , calculate the first L1 loss, that is:

[0103] ;

[0104] in, is the first loss, H1 is the normal brightness grayscale image sample The height of W1 is the normal brightness grayscale image sample The width of is the L1 norm.

[0105] Using enhanced image samples With low brightness image sample Corresponding normal brightness grayscale image samples , calculate the second L1 loss and structural similarity loss, that is:

[0106] ;

[0107] ;

[0108] in, is the second L1 loss, N is the number of color channels, The initial re-rendering network is constrained at the structural level for the structural similarity loss. To enhance the image sample With normal brightness grayscale image sample The structural similarity function between .

[0109] The structural similarity function can be expressed as:

[0110]

[0111] in, , Represent enhanced image samples The pixel mean and normal brightness grayscale image samples The pixel mean, , Represent enhanced image samples The variance of the normal brightness grayscale image sample The variance of Represents an enhanced image sample With normal brightness grayscale image sample The covariance of . and are two extremely small constant parameters used to ensure that the denominator is not zero.

[0112] Finally, the first L1 loss, the second L1 loss and the structural similarity loss are weightedly summed to obtain a comprehensive loss. The comprehensive loss is used to iteratively update the structural parameters of the initial grayscale transformation network and the initial re-rendering network. When the preset number of iterations is reached or the comprehensive loss converges, the grayscale transformation network and the re-rendering network are obtained.

[0113] like Figure 4 As shown, the first row and the second row are respectively the original image and the image enhancement result obtained by the image enhancement algorithm provided in the embodiment of the present invention. Figure 4 It can be seen that the image enhancement results are rich in details, bright in color, and have significant image features, which helps to improve the performance of high-level visual tasks.

[0114] like Figure 5 As shown, based on the above embodiment, an image enhancement device is provided in an embodiment of the present invention, including:

[0115] A grayscale image prediction module 51 is used to predict a grayscale image of an original image by applying a grayscale transformation network based on the grayscale world hypothesis theory;

[0116] An image enhancement module 52, configured to apply the original image to re-render the grayscale image based on a re-rendering network, restore the color of the grayscale image, and obtain an image enhancement result;

[0117] Among them, the re-rendering network includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer, the first feature extraction branch is used to extract the first feature of the original image, the second feature extraction branch is used to extract the second feature of the grayscale image, the dual feature fusion module is used to fuse the first feature with the second feature to obtain a fusion result, and the decoding layer is used to obtain the image enhancement result based on the fusion result.

[0118] On the basis of the above embodiment, in the image enhancement device provided in the embodiment of the present invention, the dual-feature fusion module includes an input layer, a first depth-separable convolution layer, a multi-head attention layer, a second depth-separable convolution layer and a fusion output layer connected in sequence, and the input layer is connected to the fusion output layer;

[0119] The input layer is used to receive input features;

[0120] The first depth-wise separable convolutional layer is used to perform a convolution operation on the input feature to obtain a first output result;

[0121] The multi-head attention layer is used to extract attention features based on the first output result;

[0122] The second depth-wise separable convolutional layer is used to perform a convolution operation on the attention feature to obtain a second output result;

[0123] The fusion output layer is used to fuse the second output result with the input feature to obtain an output feature.

[0124] On the basis of the above embodiment, in the image enhancement device provided in the embodiment of the present invention, the multi-head attention layer includes a flattening layer, a matrix multiplication layer and a color mapping layer;

[0125] The flattening layer is used to flatten the first output result into a one-dimensional tensor, where the one-dimensional tensor includes a query vector, a key vector, and a value vector;

[0126] The matrix multiplication layer is used to perform matrix multiplication on the query vector and the key vector to obtain a transformation matrix;

[0127] The color mapping layer is used to transform the value vector based on the transformation matrix to obtain the attention feature.

[0128] On the basis of the above embodiments, in the image enhancement device provided in the embodiments of the present invention, the convolution operations of the first depthwise separable convolution layer and the second depthwise separable convolution layer both include partial convolution operations and depthwise separable convolution operations.

[0129] On the basis of the above embodiment, in the image enhancement device provided in the embodiment of the present invention, the first feature extraction branch and the second feature extraction branch both include feature extraction layers of multiple scales, and each feature extraction layer of each scale is correspondingly connected to one of the dual feature fusion modules;

[0130] A plurality of the dual-feature fusion modules are all connected to the decoding layer.

[0131] On the basis of the above embodiment, the image enhancement device provided in the embodiment of the present invention further includes a network training module for:

[0132] Inputting the low brightness image sample into the initial grayscale transformation network to obtain the grayscale image sample output by the initial grayscale transformation network;

[0133] Inputting the low brightness image sample and the grayscale image sample into an initial re-rendering network to obtain an enhanced image sample output by the initial re-rendering network;

[0134] Calculating a first L1 loss based on the grayscale image samples and the normal brightness grayscale image samples corresponding to the low brightness image samples, and calculating a second L1 loss and a structural similarity loss based on the enhanced image samples and the normal brightness image samples corresponding to the low brightness image samples;

[0135] Based on the first L1 loss, the second L1 loss and the structural similarity loss, structural parameters of the initial grayscale transformation network and the initial re-rendering network are iteratively updated to obtain the grayscale transformation network and the re-rendering network.

[0136] Specifically, the functions of each module in the image enhancement device provided in the embodiment of the present invention correspond one-to-one to the operation flow of each step in the above method embodiment, and the effects achieved are also consistent. Please refer to the above embodiment for details, which will not be described in detail in the embodiment of the present invention.

[0137] On the basis of the above embodiment, an embodiment of the present invention further provides a robot, comprising a controller, a robot body and an image acquisition device mounted on the robot body, wherein the controller is connected to the robot body and the image acquisition device;

[0138] The image acquisition device is used to acquire the original image of the robot body during the working process, and send the original image to the controller;

[0139] The controller is used to determine the image enhancement result of the original image based on the image enhancement method based on the gray world hypothesis theory provided in the above embodiments, and control the robot body based on the image enhancement result.

[0140] Specifically, the robot can enhance the original image acquired by the image acquisition device through the controller, which can improve the visual environment of the robot operation, so that the robot equipped with the image acquisition device can work efficiently in a low-light environment, providing a strong technical guarantee for intelligent manufacturing and industrial production. In addition, the method can be applied to various types of robots, which is conducive to improving the robustness of the robot to the environment.

[0141] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor (Processor) 610, a communication interface (Communications Interface) 620, a memory (Memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the image enhancement method based on the gray world hypothesis theory provided in the above embodiments.

[0142] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0143] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image enhancement method based on the gray world hypothesis theory provided in the above embodiments.

[0144] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image enhancement method based on the gray world hypothesis theory provided in the above-mentioned embodiments.

[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image enhancement method based on gray world hypothesis theory, characterized in that: include: Based on the grayscale world hypothesis theory, the grayscale transformation network is applied to predict the grayscale image of the original image; The grayscale transformation network includes a plurality of encoders with gradually decreasing scales and a plurality of decoders with gradually increasing scales connected in sequence, and the number of scales of the encoder is the same as the number of scales of the decoder; the encoder includes a convolution activation layer, and the decoder includes a transposed convolution layer and a convolution activation layer, and the encoder is correspondingly connected to the convolution activation layer in the decoder of the same scale; Based on the re-rendering network, applying the original image to re-render the grayscale image, restoring the color of the grayscale image, and obtaining an image enhancement result; The re-rendering network includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer, the first feature extraction branch is used to extract the first feature of the original image, the second feature extraction branch is used to extract the second feature of the grayscale image, the dual feature fusion module is used to fuse the first feature with the second feature to obtain a fusion result, and the decoding layer is used to obtain the image enhancement result based on the fusion result; The dual-feature fusion module includes an input layer, a first depth-separable convolutional layer, a multi-head attention layer, a second depth-separable convolutional layer and a fusion output layer connected in sequence, and the input layer is connected to the fusion output layer; The input layer is used to receive input features; The first depth-wise separable convolutional layer is used to perform a convolution operation on the input feature to obtain a first output result; The multi-head attention layer is used to extract attention features based on the first output result; The second depth-wise separable convolutional layer is used to perform a convolution operation on the attention feature to obtain a second output result; The fusion output layer is used to fuse the second output result with the input feature to obtain an output feature.

2. The image enhancement method based on the gray world hypothesis theory according to claim 1, characterized in that: The multi-head attention layer includes a flattening layer, a matrix multiplication layer and a color mapping layer; The flattening layer is used to flatten the first output result into a one-dimensional tensor, where the one-dimensional tensor includes a query vector, a key vector, and a value vector; The matrix multiplication layer is used to perform matrix multiplication on the query vector and the key vector to obtain a transformation matrix; The color mapping layer is used to transform the value vector based on the transformation matrix to obtain the attention feature.

3. The image enhancement method based on the gray world hypothesis theory according to claim 1, characterized in that: The convolution operations of the first depthwise separable convolution layer and the second depthwise separable convolution layer both include partial convolution operations and depthwise separable convolution operations.

4. The image enhancement method based on the gray world hypothesis theory according to claim 1, characterized in that: The first feature extraction branch and the second feature extraction branch each include feature extraction layers of multiple scales, and each feature extraction layer of each scale is correspondingly connected to one of the dual feature fusion modules; A plurality of the dual-feature fusion modules are all connected to the decoding layer.

5. The image enhancement method based on the gray world hypothesis theory according to any one of claims 1 to 4, characterized in that: The grayscale transformation network and the re-rendering network are jointly trained based on the following steps: Inputting the low brightness image sample into the initial grayscale transformation network to obtain the grayscale image sample output by the initial grayscale transformation network; Inputting the low brightness image sample and the grayscale image sample into an initial re-rendering network to obtain an enhanced image sample output by the initial re-rendering network; Calculating a first L1 loss based on the grayscale image samples and the normal brightness grayscale image samples corresponding to the low brightness image samples, and calculating a second L1 loss and a structural similarity loss based on the enhanced image samples and the normal brightness image samples corresponding to the low brightness image samples; Based on the first L1 loss, the second L1 loss and the structural similarity loss, structural parameters of the initial grayscale transformation network and the initial re-rendering network are iteratively updated to obtain the grayscale transformation network and the re-rendering network.

6. An image enhancement device, characterized in that: include: A grayscale image prediction module is used to predict the grayscale image of the original image based on the grayscale world hypothesis theory and the grayscale transformation network; the grayscale transformation network includes a plurality of encoders with gradually decreasing scales and a plurality of decoders with gradually increasing scales connected in sequence, and the number of scales of the encoder is the same as the number of scales of the decoder; the encoder includes a convolution activation layer, the decoder includes a transposed convolution layer and a convolution activation layer, and the encoder is correspondingly connected to the convolution activation layer in the decoder of the same scale; An image enhancement module, used to apply the original image to re-render the grayscale image based on a re-rendering network, restore the color of the grayscale image, and obtain an image enhancement result; The re-rendering network includes a first feature extraction branch, a second feature extraction branch, a dual feature fusion module and a decoding layer, the first feature extraction branch is used to extract the first feature of the original image, the second feature extraction branch is used to extract the second feature of the grayscale image, the dual feature fusion module is used to fuse the first feature with the second feature to obtain a fusion result, and the decoding layer is used to obtain the image enhancement result based on the fusion result; The dual-feature fusion module includes an input layer, a first depth-separable convolutional layer, a multi-head attention layer, a second depth-separable convolutional layer and a fusion output layer connected in sequence, and the input layer is connected to the fusion output layer; The input layer is used to receive input features; The first depth-wise separable convolutional layer is used to perform a convolution operation on the input feature to obtain a first output result; The multi-head attention layer is used to extract attention features based on the first output result; The second depth-wise separable convolutional layer is used to perform a convolution operation on the attention feature to obtain a second output result; The fusion output layer is used to fuse the second output result with the input feature to obtain an output feature.

7. A robot, characterized in that: It includes a controller, a robot body and an image acquisition device mounted on the robot body, wherein the controller is connected to the robot body and the image acquisition device; The image acquisition device is used to acquire the original image of the robot body during the working process, and send the original image to the controller; The controller is used to determine the image enhancement result of the original image based on the image enhancement method based on the gray world hypothesis theory as described in any one of claims 1 to 5, and control the robot body based on the image enhancement result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image enhancement method based on the gray world hypothesis theory as described in any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image enhancement method based on the gray world hypothesis theory as described in any one of claims 1 to 5 is implemented.

Images