Low-illumination image enhancement method and system based on text guidance and frequency domain modulation

Abstract

The invention relates to a low-illumination image enhancement method and system based on text guidance and frequency domain modulation. Comprising the following steps: preprocessing an input image; constructing a prompt text set paired with the training data set; designing a low-illumination image enhancement network based on text guidance and frequency domain modulation; designing a loss function for parameter updating of the low-illumination image enhancement network based on text guidance and frequency domain modulation, and training the network by using a training data set and a prompt text set paired with the training data set to obtain a trained low-illumination image enhancement network based on text guidance and frequency domain modulation; and inputting an image to be detected and the prompt text set paired with the image to be detected into the trained low-illumination image enhancement network, and generating a normal-illumination image by using the trained network. According to the method, the text prompt generated by the low-illumination image is utilized, the image restoration of region perception is realized through semantic guidance, and the brightness, the color fidelity and the detail definition of the image are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image and video processing and computer vision, in particular to a low-illumination image enhancement method and system based on text guidance and frequency domain modulation. BACKGROUND

[0002] With the rapid development of image processing and computer vision technology, low-illumination image enhancement as an important image processing technology has been widely concerned by the academic and industrial circles in recent years. In low-illumination environment, due to the configuration and actual use state of the shooting device, the collected image often has problems such as poor illumination, significant noise and color distortion. In order to solve this problem, the low-illumination image enhancement technology is used to improve the brightness, contrast and detail clarity of the image, so that the enhanced image is closer to the imaging effect under normal lighting conditions, and meets the needs of human visual perception and downstream tasks. The current low-illumination image enhancement task not only needs to enhance the image to be close to the normal illumination image at the pixel level, but also needs to realize the restoration of the image region structure and the enhancement of the details, so how to realize the region-aware image restoration is an important challenge in this field.

[0003] Early low-illumination image enhancement methods mainly rely on traditional image processing technologies such as histogram equalization and Retinex theory. The core principle of the histogram equalization method is to uniformly distribute the gray histogram of the image in the entire gray scale range, so as to enhance the contrast of the image, but this method is easy to cause the problem of uneven local brightness distribution of the image; the low-illumination image is decomposed into illumination component and reflection component based on the Retinex method, which restores the edge features of the image while improving the brightness of the image, but when facing image noise and complex illumination changes, the enhancement effect of this method decreases significantly.

[0004] In recent years, methods based on deep learning have been widely studied and concerned in the field of low-illumination image enhancement. Compared with traditional methods, this method uses the powerful feature learning ability of deep neural network to restore low-illumination image more effectively. However, this kind of method mainly uses image features for global brightness and color restoration, but it fails to better solve the problem of region structure restoration and detail enhancement, resulting in problems such as detail loss and color distortion in the image region.

[0005] In the field of low-light image enhancement, in addition to exploring image features, semantic information of the text mode can also be introduced to realize semantic-guided image degradation recovery. However, most of the existing low-light image enhancement methods combined with text guide image enhancement based on the class description of positive / negative samples, but this method can only realize the global brightness adjustment of the image, and cannot realize the semantic-aware regional image recovery; some methods rely on high-quality images to generate text prompts, but in actual real scenes, it is difficult to obtain high-quality images, which limits the feasibility of its real scene application. Therefore, how to generate text prompts from existing low-light images and effectively realize semantic-guided regional-aware image recovery has become a key problem to be solved. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a low-light image enhancement method and system based on text guidance and frequency domain modulation, which generates prompt text describing the object in the image by using a visual understanding model, and establishes a precise correspondence between the prompt text and the degraded region of the image through a text-guided regional enhancement module, and then realizes the perception and recovery of the degraded region by using semantic-guided attention operation; at the same time, the frequency domain modulation module is used to realize the recovery of high-frequency details and the enhancement of low-frequency structures of the image, so that the generated enhanced image has good comprehensive performance in brightness, color fidelity and detail clarity.

[0007] To achieve the above purpose, the present application adopts the following technical solution: a low-light image enhancement method based on text guidance and frequency domain modulation, comprising the following steps:

[0008] Step A, pre-processing the input image, the pre-processing including image data pairing, cropping and flipping operation, obtaining the training data set, and processing the test image by using the same pairing, cropping and flipping operation to obtain the test data set;

[0009] Step B, constructing the prompt text set paired with the training data set and the prompt text set paired with the test data set; inputting the input low-light image and the prompt instruction into the visual understanding model to obtain the prompt text containing the description of the object in the image, then pre-processing the prompt text, including prompt text quantization and text encoding operation, to obtain the prompt text set paired with the training data set; by using the same prompt instruction, text generation method and text preprocessing process as in the training stage, the prompt text set paired with the test data set is obtained;

[0010] Step C, designing a low-light image enhancement network based on text guidance and frequency domain modulation, the low-light image enhancement network including an RGB to HVI color space conversion module, an enhancement network based on text guidance regional perception and high-low frequency joint modulation, and an HVI to RGB color space conversion module;

[0011] Step D, design a loss function for parameter updating of the low-light image enhancement network based on text guidance and frequency domain modulation in step C, and train the low-light image enhancement network based on text guidance and frequency domain modulation in step C using the training data set and the prompt text set paired therewith constructed in step B, to obtain a trained low-light image enhancement network based on text guidance and frequency domain modulation;

[0012] Step E, input the test data set and the prompt text set paired therewith into the trained low-light image enhancement network obtained in step D, to obtain a normal-illumination image.

[0013] In a preferred embodiment, step A is implemented as follows:

[0014] Step A1, pairing a normal-illumination image with a low-light image as a label image;

[0015] Step A2, uniformly and randomly cropping each low-light image and normal-illumination image to obtain an image with a size of , wherein , is the height and width of the cropped image;

[0016] Step A3, implementing image data enhancement by randomly horizontally flipping and randomly up-down flipping each low-light image and normal-illumination image, to obtain an input RGB image .

[0017] In a preferred embodiment, step B is implemented as follows:

[0018] Step B1, designing a prompt instruction for asking the visual understanding model which objects in the current input low-light image need to be brightened and restored, and the input prompt instruction is “This is a low-light image. Please provide five objects that you think need to be brightened and restored”;

[0019] Step B2, using the prompt instruction designed in step B1 as input together with each input image, and after response processing by the visual understanding model Qwen-VL-Max, obtaining English prompt text describing the objects in the image;

[0020] Step B3, converting each piece of prompt text obtained in step B2 from a string type to a tensor type, to obtain a tensor type of prompt text;

[0021] Step B4, input each piece of prompt text tensor obtained in step B3 into a pre-trained CLIP text encoder to obtain text features paired with the input image wherein represents the sequence length of the text features, represents the number of channels.

[0022] In a preferred embodiment, step C is specifically implemented as follows:

[0023] Step C1, color conversion is performed using an RGB-to-HVI color space conversion module; the RGB-to-HVI color space conversion module converts the input RGB image to the HSV color space and then to the HVI color space to obtain an intensity map and a color map ;

[0024] Step C2, an enhancement network based on text-guided region perception and high-low frequency joint modulation is designed, which consists of a feature input module, an intensity enhancement branch, a color enhancement branch, a feature output module, feature splicing, and addition operations. The intensity map obtained in step C1 is subjected to the feature input module, the intensity enhancement branch, and the feature output module to realize intensity feature region consistency enhancement under semantic guidance, and a reconstructed intensity map is obtained; the color map obtained in step C1 is subjected to the feature input module, the color enhancement branch, and the feature output module to realize color feature region consistency enhancement and detail information recovery, and a reconstructed color map is obtained; then, and are subjected to channel dimension splicing operations, and the color map obtained in step C1 is added to obtain a reconstructed HVI image , i.e.,

[0025]

[0026]

[0027] wherein, represents the feature input module, represents the intensity enhancement branch, represents the feature output module, represents the color enhancement branch, represents the splicing operation;

[0028] Step C3, color conversion is performed by using an HVI-to-RGB color space conversion module; the HVI-to-RGB color space conversion module first converts the reconstructed HVI image obtained in step C2 into an HSV color space, and then converts the HSV color space into an RGB color space, to obtain an enhanced RGB image .

[0029] In a preferred embodiment, step C2 is implemented as follows:

[0030] Step C21, a feature input module is designed; the feature input module comprises a convolution unit stacked by two convolution layers with a kernel size of 3x3 and a stride of 1, a bilinear down-sampling layer and a ReLu activation function; for the intensity image and the color image obtained in step C1, the intensity image and the color image are input into the convolution unit to obtain extracted intensity features and color features, respectively. wherein N is the number of feature channels, and D is the down-sampling rate.

[0031] Step C22, a text-guided region enhancement module is designed; the region enhancement module is used in parallel in the intensity enhancement branch and the color enhancement branch, and the same prompt text is introduced as semantic guidance, so as to realize intensity feature region consistency enhancement and color feature region consistency enhancement under semantic guidance, respectively; the region enhancement module in the intensity enhancement branch and the color enhancement branch has the same process, and is composed of a text alignment prompt generation operation, a text alignment calibration operation, an image feature enhancement operation and a text-image fusion operation; specifically, first, the region enhancement module generates prior features for calibrating prompt text through the text alignment prompt generation operation, then the text alignment calibration operation is used to realize the alignment and calibration of the text and the image features, then the image feature enhancement operation is used to realize the semantic-guided image feature enhancement, and finally, the text-image fusion operation is used to realize the fusion of the calibrated text features and the optimized image features.

[0032] Step C23, an intensity enhancement branch is designed; the intensity enhancement branch is composed of two intensity optimization modules and a text-guided region enhancement module; for the intensity features obtained in step C21, the two intensity optimization modules are used to obtain preliminarily enhanced intensity features, wherein the intensity optimization module is used to optimize the intensity features through attention and convolution operations; then, the obtained prompt text and the preliminarily enhanced intensity features are input into the text-guided region enhancement module to obtain enhanced intensity features. ​​​​​​​​​​​Input a text-guided region augmentation module to obtain the intensity features of the region augmentation. ;at last, go through Each strength optimization module obtains the final enhanced strength feature. ;

[0033] Step C24: Design the frequency domain modulation module. This module mainly consists of wavelet transform, low-frequency component modulation, high-frequency component modulation, and inverse wavelet transform. Specifically, firstly, the frequency domain modulation module decomposes the color features in the color enhancement branch into low-frequency and high-frequency components using wavelet transform. Then, it designs low-frequency component modulation to restore the global structure of the image and high-frequency component modulation to restore local details. Finally, it uses inverse wavelet transform to transform the image from frequency domain components to spatial features.

[0034] Step C25: Design the color enhancement branch; this color enhancement branch consists of... It consists of a color optimization module, two frequency domain modulation modules, and a text-guided region enhancement module; the color features obtained in step C21 are used to... The color features are initially enhanced after a color optimization module is used. The color optimization module optimizes color features through attention and convolution operations; subsequently, Input a frequency domain modulation module to obtain preliminary structural and detail-enhanced color features. ;then, Intermediate enhanced color features are obtained after a color optimization module. ;after, Input a text-guided region enhancement module to obtain region-enhanced color features. Furthermore, go through Each color optimization module obtains deeply enhanced color features. Subsequently, The second frequency domain modulation module is input to obtain color features with enhanced depth structure and detail. ;at last, After the first The color optimization module obtains the final enhanced color features. ;

[0035] Step C26: Design the feature output module; this feature output module contains two convolutional units, the first convolutional unit consisting of a convolutional kernel of size [missing information]. The convolutional unit consists of a stacked convolutional layer with a stride of 1, a bilinear upsampling layer, and a ReLU activation function. The second convolutional unit has a kernel size of [missing information]. And it consists of convolutional layers with a stride of 1; for the final enhanced intensity features obtained in step C23 and the final enhanced color features obtained in step C25 The reconstructed intensity map is obtained after passing through the two convolutional units mentioned above in sequence. and reconstructing color maps .

[0036] In a preferred embodiment, step C22 is specifically implemented as follows:

[0037] Step C221: Design the contextual difference feature extraction operation; specifically, transform local geometric difference features into high-level semantic features; let the input features of the operation be... After the operation, neighborhood difference aggregation features will be obtained. ,in This represents the dimensionality reduction factor of the channels; the operation first establishes the neighborhood relationships between features, and then uses the KNN algorithm to obtain... Each location feature is compared with the neighbor features of its k nearest neighbors. ,Will Perform a second-dimensional expansion operation and copy the features in that dimension. Next, generate center point features. Next, and Features after subtraction and First, a concatenation operation is performed along the channel dimension, followed by a reshaping operation to obtain the differential features. Finally, the differential features are reconstructed and aggregated using a convolutional kernel with a size of [missing value]. Convolutional layers with a stride of 1 obtain reconstructed features. ,exist The third dimension is maximized to obtain the neighborhood difference aggregation feature. ;

[0038]

[0039]

[0040]

[0041] in, This represents the KNN algorithm. This indicates a feature copying operation. Indicates an extension operation. This indicates a reshaping operation. Indicates the kernel size as Convolution operation, This indicates the operation of retrieving the maximum value;

[0042] Step C222, design text alignment cue generation operation; for processing the preliminary enhanced intensity feature obtained in step C23 and the intermediate enhanced color feature obtained in step C25 , generate intensity feature based alignment cue prior feature through this operation , generate color feature based alignment cue prior feature through this operation ; the text alignment cue generation operation processes in the intensity branch and the color branch are the same. In the intensity branch, first, the is inputted into the context difference feature extraction operation designed in step C221, and the output feature obtained through this operation is taken as the first level neighborhood difference aggregation feature ; then, is inputted into the context difference feature extraction operation again, and the output feature obtained through this operation is taken as the second level neighborhood difference aggregation feature ; next, is inputted into the third context difference feature extraction operation, and the output feature obtained through this operation is taken as the third level neighborhood difference aggregation feature ; subsequently, , and are respectively subjected to global max pooling to obtain , , , and the three features are spliced to obtain the multi-scale global feature ; finally, the is converted into by using linear mapping operation, and the is subjected to dimension expansion operation to obtain , and the specific process is represented by formula as follows:

[0043]

[0044]

[0045]

[0046] wherein, represents the level context difference feature extraction operation, represents the global max pooling operation, represents the dimension expansion operation, represents the linear mapping operation;

[0047] Step C223: Design a text alignment calibration operation; the input for this text alignment calibration operation in the intensity branch includes the information obtained in step C222. The initial enhanced intensity features obtained in step C23 and prompt text This is used to align and calibrate the text with intensity features, resulting in updated prompt text. The input for this operation in the color branch includes the information obtained in step C222. The intermediate enhanced color features obtained in step C25 and This is used to align and calibrate text and color features, resulting in updated prompt text. The text alignment calibration process is the same in both the intensity and color branches. In the intensity branch, a linear mapping operation is first used to... and Mapped to key features respectively and query features Using reshaping and linear mapping operations to Convert to query features ; then, , and In terms of channel dimension Each head, respectively obtained the first Size characteristics , and Based on this, text alignment is performed, and calculations are performed. , Cross-attention generation of text features aligned with image features Aligned text features are obtained through multi-head feature aggregation and linear mapping. Simultaneously, a text calibration operation is performed. and Cross-attention is calculated as input, and multi-head feature aggregation and linear mapping are performed to obtain calibrated text features. Then, text regularization is performed to balance the distribution of attention weights across different regions within the prompt text. The input for this operation is... Regularized text features are obtained through element-wise squaring of feature maps, channel summation, multi-head feature aggregation, and linear mapping. Finally, feature fusion is performed to... and and Perform a summation operation, and Perform the subtraction operation to obtain the updated prompt text. The specific process can be expressed by the following formula:

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] wherein, , and are linear mapping operations, , and are learnable parameters;

[0056] Step C224, designing image feature enhancement operations; the image feature enhancement operations in the intensity branch input contain the multi-head image features obtained in step C223 and the updated prompt text , through operations, realizing semantic-guided intensity feature optimization, obtaining optimized intensity features ; the image feature enhancement operations in the color branch input contain the multi-head image features obtained in step C223 and the updated prompt text , through operations, realizing semantic-guided color feature optimization, obtaining optimized color features ; the image feature enhancement operations in the intensity branch and the color branch have the same processing flow. In the intensity branch, first, map to key features through linear mapping, and divide into U heads and obtain the key feature ; then, and go through cross-attention and multi-head feature aggregation to obtain enhanced image features ; afterwards, perform image regularization operations, go through element-wise squaring of feature maps, summation of channel dimensions, and multi-head feature aggregation to obtain regularized image features ; finally, perform feature fusion operations, sum and , and subtract to obtain optimized intensity features ;

[0057] Step C225, design text-image fusion operation; the input of the text-image fusion operation in the intensity branch contains the key features of the updated prompt text obtained in step C223 and the multi-head updated prompt text obtained in step C224 and the optimized intensity features , realize the fusion of the calibrated text features and the optimized intensity features through operation, and obtain the region-enhanced intensity features ; the input of the operation in the color branch contains the key features of the updated prompt text obtained in step C223 and the multi-head updated prompt text obtained in step C224 and the optimized color features , realize the fusion of the calibrated text features and the optimized color features through operation, and obtain the region-enhanced color features ; the processing flow of the text-image fusion operation in the intensity branch and the color branch is the same. In the intensity branch, obtain the first head query feature through multi-head feature decomposition , , obtain the first head value feature through linear mapping and multi-head feature decomposition , , , and realize the fusion of the text features and the image features through cross-attention, and obtain the enhanced image features through multi-head feature aggregation ; then, add the reshaped to the preliminary enhanced intensity features obtained in step C23 to obtain the region-enhanced intensity features .

[0058] In a preferred embodiment, the step C24 is specifically implemented as follows:

[0059] Step C241, design wavelet transform operation; the wavelet transform operation respectively takes the preliminary enhanced color features obtained in step C25 and the depth-enhanced color features as inputs, and respectively decomposes and into low-frequency components containing global structures , high-frequency components of horizontal details , high-frequency components of vertical details and high-frequency components of diagonal details ;

[0060] Step C242, design a low-frequency component modulation operation; the low-frequency component modulation operation inputs the low-frequency component obtained in step C241 , and is used to realize the recovery of the global structure features of the image, to obtain an image structure-enhanced low-frequency component ; specifically, first, a learnable Token embedding is randomly initialized , which is used to extract and integrate the image structure information in the low-frequency component, where represents the sequence length of ; then, the low-frequency component is reshaped to obtain a reshaped low-frequency component , where ; then, an attention operation is performed, the is multiplied by the transpose of , and the attention weight is obtained after the Softmax operation, and then the weight matrix is weighted and summed with the after the linear mapping operation; subsequently, the image structure-enhanced sequence low-frequency component is obtained by adding ; finally, the is reshaped to obtain the image structure-enhanced low-frequency component ; the process is represented by the formula:

[0061]

[0062]

[0063]

[0064] wherein represents a linear mapping operation, represents a feature transpose operation;

[0065] Step C243, design a high-frequency component modulation operation; the high-frequency component modulation operation inputs the high-frequency component obtained in step C241 , and , and is used to realize the recovery of the local detail information of the image, to obtain an image detail-enhanced high-frequency component ; specifically, first, the , and are spliced in the channel dimension to obtain a high-frequency aggregated feature ; then, a learnable Token embedding is randomly initialized , which is used to extract and integrate the image detail information in the high-frequency component; subsequently, the low-frequency component Perform a reshaping operation to obtain the reshaped low-frequency components. Next, attention manipulation will be performed. and The transpose of the image is multiplied by a dot product, followed by instance normalization to suppress image noise while restoring details. Then, a softmax operation is performed to obtain attention weights, and this weight matrix is ​​then mapped to the matrix after a linear mapping operation. Perform a weighted summation; then, add the results together with the summation operation. Performing addition to obtain high-component sequences for image structure enhancement Finally, Perform reshaping operations to obtain high-frequency components for enhanced image detail. ;

[0066] Step C244: Design the inverse wavelet transform operation; the input of this inverse wavelet transform operation includes the low-frequency component obtained in step C242. and the high-frequency components obtained in step C243 It is used to transform an image from frequency domain components to spatial domain features, thereby obtaining color features that enhance image structure and detail. Specifically, firstly, High-frequency component decomposition is performed at the channel level to obtain the horizontal component with enhanced detail. Vertical components diagonal components Then, the inverse wavelet transform is used to... , , and Convert to reconstruct low-frequency components and high frequency components , and Finally, these four components are added together to obtain color features that enhance image structure and detail. .

[0067] In a preferred embodiment, step D is specifically implemented as follows:

[0068] Step D1: Design the loss function for a low-light image enhancement network based on text guidance and frequency domain modulation. The loss function The input includes an enhanced image in the RGB color space. and normal illumination image Enhanced images with HVI color space and normal illumination image ,in Depend on The loss function is calculated by step C, which is composed of L1 loss function , structural similarity loss function and edge loss function The total target loss function of the network is represented as follows:

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] wherein, and represent the balance parameters; and represent the average values of the enhanced image and the normal illumination image respectively, represents the covariance of and , and represent the variance of and respectively, and are constants to prevent the denominator from being 0; contains two convolution units, the first convolution unit is stacked by a convolution layer with a convolution kernel size of and a step size of 1 and a downsampling operation, and the second convolution unit is composed of a convolution layer with a convolution kernel size of and a step size of 1; represents the edge detail information of the image, represents the mean square error calculation;

[0076] Step D2, the training data set and the paired prompt text set obtained in steps A and B are randomly divided into several batches, each batch containing N pairs of images and the prompt text paired therewith;

[0077] Step D3, input the low-illumination image and the paired text feature , and obtain the enhanced HVI image and the RGB image , the loss is calculated using the formula in step D1 ;

[0078] Step D4, the network parameter gradient is calculated using the back propagation method according to the loss, and the network parameter is updated using the Adam optimization method;

[0079] Step D5, steps D2 to D4 are repeatedly executed in batches to obtain a low-illumination image enhancement network model based on text guidance and frequency domain modulation.

[0080] In a preferred embodiment, the step E is specifically implemented as follows:

[0081] Step E1, the low-illumination image in the test data set obtained in step A , and the prompt text in the prompt text set paired with the test data set obtained in step B are input into the designed network, and the output result of the trained network is used, wherein the RGB image is the final result.

[0082] The application also provides a low-illumination image enhancement system based on text guidance and frequency domain modulation, comprising a processor, a memory and a bus, the memory storing machine readable instructions executed by the processor;

[0083] When the system is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor as described in the low-illumination image enhancement method based on text guidance and frequency domain modulation.

[0084] Compared with the prior art, the application has the following beneficial effects: first, the low-illumination image and the corresponding prompt instruction are input into the visual understanding model to generate the prompt text describing the object in the image; then a low-illumination image enhancement network based on text guidance and frequency domain modulation is designed, which aligns and calibrates the prompt text to establish a precise correspondence between the prompt text and the degraded area of the image, and then uses the semantic guided attention operation to realize the perception and recovery of the degraded area, at the same time, the frequency domain modulation module is designed to realize the recovery of high-frequency details and the enhancement of low-frequency structure of the image, so that the generated normal illumination image has a comprehensive performance in brightness, color fidelity and detail clarity. Unlike other low-illumination image enhancement methods based on text prompts, the application uses low-illumination image to generate text prompts and effectively realizes semantic guided regional perception image recovery, avoiding the problem that existing low-illumination image enhancement methods rely on high-quality image to generate text prompts, and providing a more practical technical solution for low-illumination image enhancement. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 is an implementation flowchart of the method of the present application.

[0086] Figure 2 is a network structure diagram of the low-illumination image enhancement network based on text guidance and frequency domain modulation in an embodiment of the present application.

[0087] Figure 3 is a structure diagram of the region enhancement module based on text guidance in an embodiment of the present application.

[0088] Figure 4 is a structure diagram of the frequency domain modulation module in an embodiment of the present application. DETAILED DESCRIPTION

[0089] The present application will be further described below in conjunction with the accompanying drawings and embodiments.

[0090] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as would be understood by one of ordinary skill in the art to which the present application pertains.

[0091] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the present specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0092] The present application provides a low-illumination image enhancement method based on text guidance and frequency domain modulation, as shown in Figures 1-4 comprising the following steps:

[0093] Step A, pre-processing the input image, including image data pairing, cropping and flipping, etc., to obtain a training data set, and processing the test image using the same pairing, cropping and flipping operations to obtain a test data set;

[0094] Step B, constructing a prompt text set paired with the training data set and a prompt text set paired with the test data set. Input the low-illumination image and the prompt instruction into the general visual understanding model to obtain a prompt text containing the description of the object in the image, and then pre-process the prompt text, including prompt text quantization and text encoding, etc., to obtain a prompt text set paired with the training data set; similarly, by using the same prompt instruction, text generation method and text preprocessing process as in the training stage, a prompt text set paired with the test data set is obtained;

[0095] Step C, design a low-light image enhancement network based on text guidance and frequency domain modulation, which mainly consists of an RGB to HVI color space conversion module, an enhancement network based on text guidance region perception and high-low frequency joint modulation, and an HVI to RGB color space conversion module;

[0096] Step D, design a loss function for parameter updating of the low-light image enhancement network based on text guidance and frequency domain modulation in step C, and use the training data set constructed in step B and the prompt text set paired with it to train the low-light image enhancement network based on text guidance and frequency domain modulation in step C, to obtain a trained low-light image enhancement network based on text guidance and frequency domain modulation;

[0097] Step E, input the test data set and the prompt text set paired with it into the trained low-light image enhancement network obtained in step D to obtain a normal-illumination image.

[0098] Further, the step A comprises the following steps:

[0099] Step A1, pairing a normal-illumination image as a label image with a low-illumination image;

[0100] Step A2, uniformly and randomly cropping each low-illumination image and the normal-illumination image to obtain an image with a size of , wherein , is the height and width of the cropped image;

[0101] Step A3, using random horizontal flipping and random up-down flipping to each low-illumination image and the normal-illumination image to enhance the image data, and obtaining an input RGB image .

[0102] Further, the step B comprises the following steps:

[0103] Step B1, design a prompt instruction, which is used to ask the visual understanding model which objects in the current input low-illumination image need to be brightened and restored, and the input prompt instruction is “This is a low-light image. Please provide five objects that you think need to be brightened and restored.”;

[0104] Step B2, using the prompt instruction designed in step B1, input it together with each input image as input, after the response processing of the general-purpose visual understanding model (Qwen-VL-Max), the English prompt text describing the object in the image is obtained;

[0105] Step B3, for each prompt text obtained in step B2, convert it from string type to tensor type to obtain a prompt text in tensor type;

[0106] Step B4, input each prompt text tensor obtained in step B3 into the pre-trained CLIP text encoder to obtain the text features paired with the input image , wherein represents the sequence length of the text features, represents the number of channels.

[0107] Further, the step C includes the following steps:

[0108] Step C1, color conversion is performed using the RGB to HVI color space conversion module. The RGB to HVI color space conversion module converts the input RGB image to the HSV color space and then to the HVI color space to obtain the intensity map and the color map The color space conversion module of this step refers to the paper "HVI: A New Color Space for Low-light Image Enhancement" (Q Yan, Y Feng, C Zhang, et al., in Proceedings of the Computer Vision and Pattern Recognition, 2025, 5678-5687.);

[0109] Step C2, design an enhancement network based on text-guided region perception and high-low frequency joint modulation, which consists of a feature input module, an intensity enhancement branch, a color enhancement branch, a feature output module, a feature splicing, and an addition operation. The intensity map obtained in step C1 is input into the feature input module, the intensity enhancement branch, and the feature output module to realize intensity feature region consistency enhancement under semantic guidance and obtain the reconstructed intensity map ; the color map obtained in step C1 is input into the feature input module, the color enhancement branch, and the feature output module to realize color feature region consistency enhancement and detail information recovery and obtain the reconstructed color map ; then, and performing channel dimension concatenation operation and color map obtained in step C1 performing addition operation to obtain reconstructed HVI image i.e.

[0110]

[0111]

[0112] wherein, denotes a feature input module, denotes an intensity enhancement branch, denotes a feature output module, denotes a color enhancement branch, denotes a concatenation operation;

[0113] Step C3, color conversion is performed by using HVI to RGB color space conversion module. The HVI to RGB color space conversion module first converts the reconstructed HVI image obtained in step C2 to HSV color space, and then to RGB color space to obtain enhanced RGB image This step color space conversion module refers to the paper "HVI: A New Color Space for Low-light Image Enhancement" (Q Yan, Y Feng, C Zhang et al., in Proceedings of the Computer Vision and Pattern Recognition, 2025, 5678-5687.).

[0114] Further, the step C2 includes the following steps:

[0115] Step C21, a feature input module is designed. The module contains a convolution unit which is stacked by two convolution layers with kernel size and stride of 1, a bilinear down-sampling layer and a ReLu activation function. For the intensity map and color map obtained in step C1, the extracted intensity feature and color feature are obtained after inputting them into the above-mentioned convolution unit, respectively, wherein is the number of feature channels, is the down-sampling rate;

[0116] Step C22, design a text-guided region enhancement module. This module is used in parallel in the intensity enhancement branch and the color enhancement branch, and the same prompt text is introduced as semantic guidance, and the operation is used to realize the consistency enhancement of the intensity feature region and the color feature region under semantic guidance, respectively; The module is consistent in the intensity enhancement branch and the color enhancement branch, mainly composed of a text calibration prompt generation operation, a text alignment calibration operation, an image feature enhancement operation and a text image fusion operation. Specifically, first, the module generates the prior feature of the calibration prompt text through the text calibration prompt generation operation, then the text alignment calibration operation is used to realize the alignment and calibration of the text and the image feature, then the image feature enhancement operation is used to realize the semantic guided image feature enhancement, and finally, the text image fusion operation is used to realize the fusion of the calibrated text feature and the optimized image feature;

[0117] Step C23, design an intensity enhancement branch. This branch is composed of an intensity optimization module and a text-guided region enhancement module. For the intensity feature obtained in step C21, two intensity optimization modules are used to obtain the preliminary enhanced intensity feature , wherein the role of the intensity optimization module is to optimize the intensity feature through attention and convolution operations; then and the prompt text obtained in step B4 are input into a text-guided region enhancement module to obtain a region-enhanced intensity feature ; finally, after an intensity optimization module, the final enhanced intensity feature is obtained;

[0118] Step C24, design a frequency domain modulation module. This module is mainly composed of a wavelet transform operation, a low-frequency component modulation operation, a high-frequency component modulation operation and an inverse wavelet transform operation. Specifically, first, the module decomposes the color feature in the color enhancement branch into low-frequency components and high-frequency components through wavelet transform; then, the low-frequency component modulation operation is designed to realize the recovery of the global structure of the image, and the high-frequency component modulation operation is designed to realize the recovery of the local details of the image; finally, the wavelet inverse transform operation is used to convert the image from the frequency domain component to the spatial dimension feature;

[0119] Step C25, design a color enhancement branch. This branch is composed of a color optimization module, two frequency domain modulation modules and a text-guided region enhancement module. The color feature obtained in step C21 is input into a color optimization module to obtain a preliminary enhanced color feature The color optimization module optimizes color features through attention and convolution operations; subsequently, Input a frequency domain modulation module to obtain preliminary structural and detail-enhanced color features. ;then, Intermediate enhanced color features are obtained after a color optimization module. ;after, Input a text-guided region enhancement module to obtain region-enhanced color features. Furthermore, go through Each color optimization module obtains deeply enhanced color features. Subsequently, The second frequency domain modulation module is input to obtain color features with enhanced depth structure and detail. ;at last, After the first The color optimization module obtains the final enhanced color features. ;

[0120] Step C26: Design the feature output module. This module contains two convolutional units. The first convolutional unit consists of a convolutional kernel of size [missing information]. The convolutional unit consists of a stacked convolutional layer with a stride of 1, a bilinear upsampling layer, and a ReLU activation function. The second convolutional unit has a kernel size of [missing information]. It consists of convolutional layers with a stride of 1. For the final enhanced intensity features obtained in step C23... and the final enhanced color features obtained in step C25 The reconstructed intensity map is obtained after passing through the two convolutional units mentioned above in sequence. and reconstructing color maps .

[0121] Further, step C22 includes the following steps:

[0122] Step C221: Design the contextual difference feature extraction operation. This operation transforms local geometric difference features into high-level semantic features; let the input features of this operation be... This operation will yield neighborhood difference aggregation features. This operation first establishes neighborhood relationships between features, using the KNN algorithm to obtain... Each location feature is compared with the neighbor features of its k nearest neighbors. ,Will Perform a second-dimensional expansion operation and copy the features in that dimension. Next, generate center point features. Next, and Features after subtraction and First, a concatenation operation is performed along the channel dimension, followed by a reshaping operation to obtain the differential features. Finally, the differential features are reconstructed and aggregated using a convolutional kernel with a size of [missing value]. Convolutional layers with a stride of 1 obtain reconstructed features. ,exist The third dimension is maximized to obtain the neighborhood difference aggregation feature. ;

[0123]

[0124]

[0125]

[0126] in, This represents the KNN algorithm. This indicates a feature copying operation. Indicates an extension operation. This indicates a reshaping operation. Indicates the kernel size as Convolution operation, This indicates the operation of retrieving the maximum value;

[0127] Step C222: Design text calibration prompt generation operation. This operation is used to process the preliminary enhanced intensity features obtained in step C23. and the intermediate enhanced color features obtained in step C25 , This operation generates calibration cue prior features based on intensity characteristics. , This operation generates prior features for calibration prompts based on color characteristics. The text calibration prompt generation process is the same in both the intensity and color branches. In the intensity branch, firstly... As input features, the contextual difference feature extraction operation designed in step C221 is used to obtain output features, which serve as the first-level neighborhood difference aggregation features. ;after, The input features are processed again through contextual difference feature extraction, and the resulting output features serve as the second-level neighborhood difference aggregation features. ;then, After the third contextual difference feature extraction operation, the resulting output features are used as the third-level neighborhood difference aggregation features. ; then, , and Perform global max pooling respectively to obtain , , The three features are then concatenated to obtain multi-scale global features. Finally, linear mapping operations are used to... Convert to ,right Perform a dimension expansion operation to obtain The specific process can be expressed by the following formula:

[0128]

[0129]

[0130]

[0131] in, Indicates the first Level contextual differential feature extraction operation, This represents the global max pooling operation. This indicates a dimension expansion operation. This represents a linear mapping operation;

[0132] Step C223: Design the text alignment calibration operation. The input for this operation in the intensity branch includes the information obtained in step C222. The initial enhanced intensity features obtained in step C23 and the prompt text obtained in step B4 This is used to align and calibrate the text with intensity features, resulting in updated prompt text. The input for this operation in the color branch includes the information obtained in step C222. The intermediate enhanced color features obtained in step C25 and This is used to align and calibrate text and color features, resulting in updated prompt text. The text alignment calibration process is the same in both the intensity and color branches. In the intensity branch, a linear mapping operation is first used to... and Mapped to key features respectively and query features Using reshaping and linear mapping operations to Convert to query features ; then, , and In terms of channel dimension The first head, respectively, obtains the first ; on this basis, the text alignment operation is performed, the cross attention of is calculated to generate the text feature aligned with the image feature, the aligned text feature is obtained through multi-head feature aggregation and linear mapping; at the same time, the text calibration operation is performed, the cross attention is calculated with as input, and the calibrated text feature is obtained through multi-head feature aggregation and linear mapping; then, the text regularization operation is performed to balance the distribution of attention weights of different regions within the prompt text word, the input of the operation is , and the regularization text feature is obtained through element-wise squaring of the feature map, channel summation, multi-head feature aggregation and linear mapping; finally, the feature fusion operation is performed, and is summed with , and is subtracted to obtain the updated prompt text , and the specific process is represented by the formula:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] wherein are linear mapping operations, are learnable parameters;

[0141] Step C224, design the image feature enhancement operation. The input of the operation in the intensity branch contains the multi-head image feature obtained in step C223 and the updated prompt text ​​​​​​​​​​This operation enables semantically guided strength feature optimization, resulting in optimized strength features. The color branch's input includes the multi-head image features obtained in step C223. and update prompt text This operation enables semantically guided color feature optimization, resulting in optimized color features. The image feature enhancement processes in the intensity and color branches are the same. In the intensity branch, the image feature enhancement is first performed using a linear mapping... Mapping to key features and will Divide into U heads and obtain the first Head key features ;then, and Enhanced image features are obtained through cross-attention and multi-head feature aggregation. Next, image regularization is performed. Regularized image features are obtained through multi-head feature aggregation by squaring each element of the feature map and summing the channel dimensions. Finally, feature fusion is performed to... and Perform a summation operation, and Perform a subtraction operation to obtain the optimized intensity features. ;

[0142] Step C225: Design the text-image fusion operation. The input to this operation in the intensity branch includes the update prompt text obtained in step C223. Key features of the multi-head update hint text obtained in C224 and optimized intensity features This operation achieves the fusion of calibrated text features and optimized intensity features to obtain region-enhanced intensity features. The input for this operation in the color branch includes the update prompt text obtained in step C223. Key features of the multi-head update hint text obtained in C224 and optimized color features This operation achieves the fusion of calibrated text features and optimized color features, resulting in region-enhanced color features. The text-image fusion process is the same in both the intensity and color branches. In the intensity branch, The first is obtained through multi-head eigenvalue decomposition. Individual query features , The first is obtained through linear mapping and multi-head eigenvalue decomposition. Individual value characteristics , 、 and The fusion of text features and image features is achieved through cross attention, and enhanced image features are obtained through multi-head feature aggregation ; then, the reshaped is added to the preliminary enhanced intensity features obtained in step C23 to obtain the region-enhanced intensity features .

[0143] Further, the step C24 includes the following steps:

[0144] Step C241, design a wavelet transform operation. This operation takes the preliminary enhanced color features and depth-enhanced color features obtained in step C25 as input, and uses wavelet transform to decompose and into low-frequency components containing global structures, high-frequency components containing horizontal details, high-frequency components containing vertical details, and high-frequency components ;

[0145] Step C242, design a low-frequency component modulation operation. This operation inputs the low-frequency components obtained in step C241, and is used to restore the image global structure features to obtain image structure-enhanced low-frequency components ; Specifically, first, the learnable Token embedding is randomly initialized to extract and integrate the image structure information in the low-frequency components, where denotes the sequence length of ; then, the low-frequency components are reshaped to obtain reshaped low-frequency components , where ; next, the attention operation is performed, the dot product of and the transpose of is performed, and the attention weights are obtained after the Softmax operation, then the weight matrix is weighted summed with the linearly mapped ; subsequently, the sequence low-frequency components are obtained by adding to obtain image structure-enhanced sequence low-frequency components ; finally, the reshaping operation is performed on to obtain image structure-enhanced low-frequency components . This process is represented by the formula:

[0146]

[0147]

[0148]

[0149] wherein, represents a linear mapping operation, represents a feature transposition operation;

[0150] Step C243, design a high-frequency component modulation operation. The operation input contains the high-frequency component , and obtained in step C241, for realizing the recovery of image local detail information, obtaining the image detail-enhanced high-frequency component ; Specifically, first, concatenate , and in the channel dimension to obtain the high-frequency aggregated feature ; Then, randomly initialize the learnable Token embedding for extracting and integrating image detail information in the high-frequency component; Subsequently, reshape the low-frequency component to obtain the reshaped low-frequency component ; Next, perform attention operation, point multiply with the transpose of , and then perform instance normalization operation for image detail recovery while suppressing image noise; Subsequently, obtain the attention weight after using the Softmax operation, and perform weighted summation with the weight matrix and the linear mapping operation ; Then, add to obtain the image structure-enhanced sequence high component ; Finally, reshape to obtain the image detail-enhanced high-frequency component ;

[0151] Step C244, design a wavelet inverse transform operation. The operation input contains the low-frequency component obtained in step C242 and the high-frequency component obtained in step C243, for converting the image from the frequency domain dimension component to the spatial dimension feature, obtaining the image structure and detail-enhanced color feature ; Specifically, first, perform high-frequency component splitting operation on in the channel dimension to obtain the detail-enhanced horizontal component , vertical component diagonal components Then, the inverse wavelet transform is used to... , , and Convert to reconstruct low-frequency components and high frequency components , and Finally, these four components are added together to obtain color features that enhance image structure and detail. .

[0152] Further, step D includes the following steps:

[0153] Step D1: Design the loss function for a low-light image enhancement network based on text guidance and frequency domain modulation. The loss function takes an enhanced image containing the RGB color space as input. and normal illumination image Enhanced images with HVI color space and normal illumination image ,in Depend on The loss function is calculated through step C1 and is derived from the L1 loss function. Structural similarity loss function and edge loss function Composition, the network's overall objective loss function It is expressed as follows:

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160] in, and Indicates the balance parameters; and These represent enhanced images. and normal illumination image The average value, express and covariance, and respectively represent and variance, and are constants to prevent the denominator from being 0; contains two convolution units, the first convolution unit is stacked by a convolution layer with a convolution kernel size of and a step size of 1 and a downsampling operation, and the second convolution unit is composed of a convolution layer with a convolution kernel size of and a step size of 1; represents the edge detail information of the image, represents the mean square error calculation;

[0161] Step D2, the training data set and the paired prompt text set obtained in steps A and B are randomly divided into several batches, each batch containing N pairs of images and the prompt text paired with them;

[0162] Step D3, input the low-light image and the paired text feature , after the low-light image enhancement network based on text guidance and frequency domain modulation in step C, the enhanced HVI image and the RGB image , the loss is calculated using the formula in step D1;

[0163] Step D4, according to the loss, the network parameter gradient is calculated using the back propagation method, and the network parameter is updated using the Adam optimization method;

[0164] Step D5, steps D2 to D4 are repeatedly executed in batches to obtain a low-light image enhancement network model based on text guidance and frequency domain modulation.

[0165] Further, the step E includes the following steps:

[0166] Step E1, input the low-light image in the test data set obtained in step A, and the prompt text in the prompt text set paired with the test data set obtained in step B into the designed network, and output the results using the trained network, wherein the RGB image is the final result.

[0167] The above is the preferred embodiment of the present application, any changes made according to the technical solutions of the present application, as long as the generated function does not exceed the scope of the technical solutions of the present application, all belong to the protection scope of the present application.

Claims

1. A low-light image enhancement method based on text guidance and frequency domain modulation, characterized in that, Includes the following steps: Step A: Preprocess the input image, including image data pairing, cropping, and flipping operations, to obtain the training dataset. The same pairing, cropping, and flipping operations are used to process the test image to obtain the test dataset. Step B: Construct a set of prompt texts paired with the training dataset and a set of prompt texts paired with the test dataset; input the low-light image and prompt instructions into the Tongyi Qianwen visual understanding model to obtain prompt texts containing descriptions of objects in the image; then preprocess the prompt texts, including prompt text tensorization and text encoding operations, to obtain a set of prompt texts paired with the training dataset; by using the same prompt instructions, text generation methods, and text preprocessing procedures as in the training phase, obtain a set of prompt texts paired with the test dataset. Step C: Design a low-light image enhancement network based on text guidance and frequency domain modulation. The low-light image enhancement network includes an RGB to HVI color space conversion module, an enhancement network based on text guidance, region perception and high and low frequency joint modulation, and an HVI to RGB color space conversion module. Step D: Design a loss function for parameter updating of the low-light image enhancement network based on text guidance and frequency domain modulation in Step C, and train the low-light image enhancement network based on text guidance and frequency domain modulation described in Step C using the training dataset constructed in Step B and its paired prompt text set to obtain the trained low-light image enhancement network based on text guidance and frequency domain modulation. Step E: Input the test dataset and its paired prompt text set into the trained low-light image enhancement network obtained in step D to obtain a normal-light image.

2. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 1, characterized in that, The specific implementation steps of step A are as follows: Step A1: Pair the normal illumination image as the label image with the low illumination image; Step A2: Randomly crop each low-light image and the normal-light image in the same way to obtain a size of [size missing]. The image, in which, , It involves cropping the height and width of the image; Step A3: Randomly flip each low-light image and normal-light image horizontally and vertically to enhance the image data and obtain the input RGB image. .

3. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 1, characterized in that, The specific implementation steps of step B are as follows: Step B1: Design a prompt instruction. The prompt instruction is used to ask the visual understanding model which objects in the currently input low-light image need to be brightened and restored. The specific prompt instruction is "This is a low-light image. Please provide five objects that you think need to be brightened and restored". Step B2: Using the prompt instructions designed in Step B1, along with each input image, as input, and after processing by the Qwen-VL-Max visual understanding model, obtain the English prompt text describing the objects in the image; Step B3: For each prompt text obtained in step B2, convert it from a string type to a tensor type to obtain a prompt text of tensor type; Step B4: Input each prompt text tensor obtained in Step B3 into the pre-trained CLIP text encoder to obtain text features paired with the input image. ,in The sequence length representing text features. Indicates the number of channels.

4. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 1, characterized in that, The specific implementation steps of step C are as follows: Step C1: Perform color conversion using the RGB to HVI color space conversion module; the RGB to HVI color space conversion module will convert the input RGB image... Convert to HSV color space and then to HVI color space to obtain the intensity map. and color chart ; Step C2: Design an enhancement network based on text-guided region perception and high / low frequency joint modulation. This enhancement network consists of a feature input module, an intensity enhancement branch, a color enhancement branch, a feature output module, feature concatenation, and an addition operation. The intensity map obtained in step C1 is then processed... After passing through the feature input module, intensity enhancement branch, and feature output module, semantically guided consistency enhancement of intensity feature regions is achieved, resulting in a reconstructed intensity map. ; The color map obtained in step C1 The feature input module, color enhancement branch, and feature output module work together to enhance the consistency of color feature regions and restore detail information, resulting in a reconstructed color map. After that, and Perform channel-dimensional stitching and combine it with the color map obtained in step C1. Perform an addition operation to obtain the reconstructed HVI image. ,Right now: in, Indicates the feature input module, Indicates a branch with increased strength. Indicates the feature output module. Indicates the color enhancement branch, Indicates a splicing operation; Step C3: Perform color conversion using the HVI to RGB color space conversion module; the HVI to RGB color space conversion module first converts the reconstructed HVI image obtained in step C2. Convert to HSV color space, then convert to RGB color space to obtain an enhanced RGB image. .

5. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 4, characterized in that, The specific implementation steps of step C2 are as follows: Step C21: Design the feature input module; The feature input module contains a convolutional unit, which consists of two convolutional kernels of size [size missing]. Furthermore, it consists of a convolutional layer with a stride of 1, a bilinear downsampling layer, and a ReLU activation function stacked together; for the intensity map obtained in step C1 and color chart The extracted intensity features are obtained by inputting them into the aforementioned convolutional units. and color characteristics ,in The number of feature channels, This is the downsampling factor; Step C22: Design a text-guided area enhancement module; This region enhancement module is used in parallel in the intensity enhancement branch and the color enhancement branch, introducing the same prompt text as semantic guidance. This operation is used to achieve consistency enhancement of intensity feature regions and consistency enhancement of color feature regions under semantic guidance, respectively. The region enhancement module follows the same process in the intensity enhancement and color enhancement branches, consisting of text calibration prompt generation, text alignment calibration, image feature enhancement, and text-image fusion operations. Specifically, firstly, the region enhancement module generates prior features for calibration prompt text through the text calibration prompt generation operation. Then, it uses the text alignment calibration operation to align and calibrate the text and image features. Next, it uses the image feature enhancement operation to achieve semantically guided image feature enhancement. Finally, it uses the text-image fusion operation to fuse the calibrated text features with the optimized image features. Step C23: Design strength-enhancing branches; This strength-enhancing branch is composed of It consists of an intensity optimization module and a text-guided region enhancement module; for the intensity features obtained in step C21 After two strength optimization modules, the initial enhanced strength characteristics are obtained. The intensity optimization module optimizes intensity features through attention and convolution operations; subsequently... and the obtained prompt text Input a text-guided region augmentation module to obtain the intensity features of the region augmentation. ;at last, go through Each strength optimization module obtains the final enhanced strength feature. ; Step C24: Design the frequency domain modulation module. This module mainly consists of wavelet transform, low-frequency component modulation, high-frequency component modulation, and inverse wavelet transform. Specifically, firstly, the frequency domain modulation module decomposes the color features in the color enhancement branch into low-frequency and high-frequency components using wavelet transform. Then, it designs low-frequency component modulation to restore the global structure of the image and high-frequency component modulation to restore local details. Finally, it uses inverse wavelet transform to transform the image from frequency domain components to spatial features. Step C25: Design the color enhancement branch; This color enhancement branch is composed of It consists of a color optimization module, two frequency domain modulation modules, and a text-guided region enhancement module; the color features obtained in step C21 are used to... The color features are initially enhanced after a color optimization module is used. The color optimization module optimizes color features through attention and convolution operations; subsequently, Input a frequency domain modulation module to obtain preliminary structural and detail-enhanced color features. ;then, Intermediate enhanced color features are obtained after a color optimization module. ;after, Input a text-guided region enhancement module to obtain region-enhanced color features. Furthermore, go through Each color optimization module obtains deeply enhanced color features. ; Subsequently, The second frequency domain modulation module is input to obtain color features with enhanced depth structure and detail. ;at last, After the first The color optimization module obtains the final enhanced color features. ; Step C26: Design the feature output module; This feature output module contains two convolutional units. The first convolutional unit consists of a convolutional kernel of size [size missing]. The convolutional unit consists of a stacked convolutional layer with a stride of 1, a bilinear upsampling layer, and a ReLU activation function. The second convolutional unit has a kernel size of [missing information]. And it consists of convolutional layers with a stride of 1; for the final enhanced intensity features obtained in step C23 and the final enhanced color features obtained in step C25 The reconstructed intensity map is obtained after passing through the two convolutional units mentioned above in sequence. and reconstructing color maps .

6. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 5, characterized in that, The specific implementation steps of step C22 are as follows: Step C221: Design the contextual difference feature extraction operation; specifically, transform local geometric difference features into high-level semantic features; let the input features of the operation be... After the operation, neighborhood difference aggregation features will be obtained. ,in This represents the dimensionality reduction factor of the channels; the operation first establishes the neighborhood relationships between features, and then uses the KNN algorithm to obtain... Each location feature is compared with the neighbor features of its k nearest neighbors. ,Will Perform a second-dimensional expansion operation and copy the features in that dimension. Next, generate center point features. Next, and Features after subtraction and First, a concatenation operation is performed along the channel dimension, followed by a reshaping operation to obtain the differential features. Finally, the differential features are reconstructed and aggregated using a convolutional kernel with a size of [missing value]. Convolutional layers with a stride of 1 obtain reconstructed features. ,exist The third dimension is maximized to obtain the neighborhood difference aggregation feature. ; in, This represents the KNN algorithm. This indicates a feature copying operation. Indicates an extension operation. This indicates a reshaping operation. Indicates the kernel size as Convolution operation, This indicates the operation of retrieving the maximum value; Step C222: Design a text calibration prompt generation operation; used to process the preliminary enhanced intensity features obtained in step C23. and the intermediate enhanced color features obtained in step C25 , This operation generates calibration cue prior features based on intensity characteristics. , This operation generates prior features for calibration prompts based on color characteristics. The text calibration prompt generation process is the same in both the intensity and color branches; in the intensity branch, firstly... As input features, the contextual difference feature extraction operation designed in step C221 is used to obtain output features, which serve as the first-level neighborhood difference aggregation features. ;after, The input features are processed again through contextual difference feature extraction, and the resulting output features serve as the second-level neighborhood difference aggregation features. ;then, After the third contextual difference feature extraction operation, the resulting output features are used as the third-level neighborhood difference aggregation features. ; then, , and Perform global max pooling respectively to obtain , , The three features are then concatenated to obtain multi-scale global features. Finally, linear mapping operations are used to... Convert to ,right Perform a dimension expansion operation to obtain The specific process can be expressed by the following formula: in, Indicates the first Level contextual differential feature extraction operation, This represents the global max pooling operation. This indicates a dimension expansion operation. This represents a linear mapping operation; Step C223: Design a text alignment calibration operation; the input for this text alignment calibration operation in the intensity branch includes the information obtained in step C222. The initial enhanced intensity features obtained in step C23 and prompt text This is used to align and calibrate the text with intensity features, resulting in updated prompt text. The input for this operation in the color branch includes the information obtained in step C222. The intermediate enhanced color features obtained in step C25 and This is used to align and calibrate text and color features, resulting in updated prompt text. The text alignment calibration process is the same in both the intensity and color branches; in the intensity branch, a linear mapping operation is first used to... and Mapped to key features respectively and query features Using reshaping and linear mapping operations to Convert to query features ; then, , and In terms of channel dimension Each head, respectively obtained the first Size characteristics , and Based on this, text alignment is performed, and calculations are performed. , Cross-attention generation of text features aligned with image features Aligned text features are obtained through multi-head feature aggregation and linear mapping. Simultaneously, a text calibration operation is performed. and Cross-attention is calculated as input, and multi-head feature aggregation and linear mapping are performed to obtain calibrated text features. Then, text regularization is performed to balance the distribution of attention weights across different regions within the prompt text. The input for this operation is... Regularized text features are obtained through element-wise squaring of feature maps, channel summation, multi-head feature aggregation, and linear mapping. Finally, feature fusion is performed to... and and Perform a summation operation, and Perform the subtraction operation to obtain the updated prompt text. The specific process can be expressed by the following formula: in, , and For linear mapping operations, , and These are learnable parameters; Step C224: Design an image feature enhancement operation; the input of this image feature enhancement operation in the intensity branch includes the multi-head image features obtained in step C223. and update prompt text By performing operations to optimize semantically guided intensity features, the optimized intensity features are obtained. The operation input in the color branch includes the multi-head image features obtained in step C223. and update prompt text By performing operations to achieve semantically guided color feature optimization, optimized color features are obtained. The image feature enhancement processes in the intensity and color branches are the same; in the intensity branch, the image feature enhancement is first performed through linear mapping. Mapping to key features and will Divide into U heads and obtain the first Head key features ;then, and Enhanced image features are obtained through cross-attention and multi-head feature aggregation. Next, image regularization is performed. Regularized image features are obtained through multi-head feature aggregation by squaring each element of the feature map and summing the channel dimensions. Finally, feature fusion is performed to... and Perform a summation operation, and Perform a subtraction operation to obtain the optimized intensity features. ; Step C225: Design a text-image fusion operation; the input of this text-image fusion operation in the intensity branch includes the update prompt text obtained in step C223. Key features of the multi-head update hint text obtained in C224 and optimized intensity features By performing operations to fuse calibrated text features with optimized intensity features, region-enhanced intensity features are obtained. The operation input in the color branch includes the update prompt text obtained in step C223. Key features of the multi-head update hint text obtained in C224 and optimized color features By performing operations to fuse calibrated text features with optimized color features, region-enhanced color features are obtained. The text-image fusion process is the same in both the intensity and color branches; in the intensity branch, The first is obtained through multi-head eigenvalue decomposition. Individual query features , The first is obtained through linear mapping and multi-head eigenvalue decomposition. Individual value characteristics , , and Text and image features are fused through cross-attention, and enhanced image features are obtained through multi-head feature aggregation. Afterwards, the reshaping operation will be performed. The initial enhanced intensity features obtained in step C23 Perform an addition operation to obtain the region enhancement intensity features. .

7. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 5, characterized in that, The specific implementation steps of step C24 are as follows: Step C241: Design wavelet transform operation; this wavelet transform operation will transform the initially enhanced color features obtained in step C25. and depth-enhanced color features As input, wavelet transform is used to respectively and Decomposed into low-frequency components containing global structure High-frequency components of horizontal details High-frequency components of vertical details and high-frequency components of diagonal details ; Step C242: Design low-frequency component modulation operation; The low-frequency component modulation operation input includes the low-frequency component obtained in step C241. This low-frequency component modulation operation is used to restore the global structural features of the image, obtaining image structure-enhanced low-frequency components. Specifically, firstly, the learnable token embedding is randomly initialized. It is used to extract and integrate image structure information from low-frequency components, where express The sequence length; then, the low-frequency components Perform a reshaping operation to obtain the reshaped low-frequency components. ,in Next, attention manipulation will be performed. and The transpose of the matrix is ​​multiplied by a dot product, and then subjected to a softmax operation to obtain the attention weights. This weight matrix is ​​then mapped to the matrix after a linear mapping operation. Perform a weighted summation; Subsequently, through the addition operation and Performing addition to obtain the image structure enhancement sequence low-frequency components Finally, Perform reshaping operations to obtain low-frequency components for image structure enhancement. The process can be expressed by the following formula: in, This represents a linear mapping operation. Indicates the feature transpose operation; Step C243: Design a high-frequency component modulation operation; the input of this high-frequency component modulation operation includes the high-frequency components obtained in step C241. , and It is used to restore local detail information of an image and obtain high-frequency components for image detail enhancement. Specifically, firstly, , and Perform a concatenation operation along the channel dimension to obtain high-frequency aggregated features. Afterwards, randomly initialize learnable token embeddings. Used to extract and integrate image detail information from high-frequency components; subsequently, low-frequency components... Perform a reshaping operation to obtain the reshaped low-frequency components. Next, attention manipulation will be performed. and The transpose of the image is multiplied by a dot product, followed by instance normalization to suppress image noise while restoring details. Then, a softmax operation is performed to obtain attention weights, and this weight matrix is ​​then mapped to the matrix after a linear mapping operation. Perform a weighted summation; then, add the results together with the summation operation. Performing addition to obtain high-component sequences for image structure enhancement Finally, Perform reshaping operations to obtain high-frequency components for enhanced image detail. ; Step C244: Design the inverse wavelet transform operation; the input of this inverse wavelet transform operation includes the low-frequency component obtained in step C242. and the high-frequency components obtained in step C243 It is used to transform an image from frequency domain components to spatial domain features, thereby obtaining color features that enhance image structure and detail. Specifically, firstly, High-frequency component decomposition is performed at the channel level to obtain the horizontal component with enhanced detail. Vertical components diagonal components Then, the inverse wavelet transform is used to... , , and Convert to reconstruct low-frequency components and high frequency components , and Finally, these four components are added together to obtain color features that enhance image structure and detail. .

8. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 1, characterized in that, The specific implementation steps of step D are as follows: Step D1: Design the loss function for a low-light image enhancement network based on text guidance and frequency domain modulation. The loss function The input includes an enhanced image in the RGB color space. and normal illumination image Enhanced images with HVI color space and normal illumination image ,in Depend on The loss function, calculated via step C, is derived from the L1 loss function. Structural similarity loss function and edge loss function Composition, the network's overall objective loss function It is expressed as follows: in, and Indicates the balance parameters; and These represent enhanced images. and normal illumination image The average value, express and covariance, and They represent and variance and It is a constant to prevent the denominator from being zero; It contains two convolutional units, the first of which consists of a convolutional kernel of size 1. Furthermore, the convolutional layer with a stride of 1 is stacked with a downsampling operation, and the second convolutional unit consists of a convolutional kernel with a size of 1. It consists of convolutional layers with a stride of 1; Represents edge detail information of the image. This indicates the calculation of mean square error; Step D2: Randomly divide the training dataset and paired prompt text set obtained in Step A and Step B into several batches. Each batch contains N pairs of images and their paired prompt text. Step D3: Input low-light image and paired text features The enhanced HVI image is obtained after passing through the low-light image enhancement network based on text guidance and frequency domain modulation in step C. and RGB images Calculate the loss using the formula in step D1. ; Step D4: Calculate the gradient of the network parameters using the backpropagation method based on the loss, and update the network parameters using the Adam optimization method; Step D5: Repeat steps D2 to D4 in batches to obtain a low-light image enhancement network model based on text guidance and frequency domain modulation.

9. The low-light image enhancement method based on text guidance and frequency domain modulation according to claim 1, characterized in that, The specific implementation steps of step E are as follows: Step E1: Extract the low-light images from the test dataset obtained in Step A. And the prompt text in the prompt text set paired with the test dataset obtained in step B. The input is fed into the designed network, and the trained network outputs the result, including the RGB image. This is the final result.

10. A low-light image enhancement system based on text guidance and frequency domain modulation, comprising a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executed by the processor; characterized in that, When the system is running, the processor and the memory communicate via a bus, and the machine-readable instructions are executed by the processor as described in any one of claims 1 to 9.

Images