End-to-end low-illumination image detection method based on frequency domain enhancement and related equipment

Abstract

The invention provides an end-to-end low-illumination image detection method based on frequency domain enhancement and related equipment. The method comprises the following steps: carrying out fast Fourier transform on a low-illumination training image by utilizing a Fourier enhancement sub-network in a frequency domain enhancement target detection network to obtain a frequency domain characteristic graph; performing complex convolution on the frequency domain feature map, continuously performing multiple times of frequency domain processing on a result of the complex convolution to obtain a frequency spectrum feature map, and performing dimensionality reduction and range constraint on the frequency spectrum feature map to obtain a second frequency domain conversion map; amplifying the amplitude of the frequency domain feature map by dividing the amplitude of the frequency domain feature map by the second frequency domain conversion map, and performing inverse fast Fourier transform by using the phase and the amplified amplitude to obtain an enhanced image; and using a preset joint loss function in the target detection sub-network to complete end-to-end training on the frequency domain enhanced target detection network.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of image recognition, and in particular to an end-to-end low-light image detection method based on frequency domain enhancement and related equipment. Background Art

[0002] Low-light image target detection algorithms based on deep learning generally have problems such as low accuracy, high computational complexity, and insufficient model generalization ability.

[0003] Specifically, many target detection algorithms perform well on images under normal lighting conditions, but cannot achieve ideal results in low-light or dark-light environments, such as nighttime autonomous driving, face detection, security monitoring and other applications. This is because the image quality is significantly degraded in dark-light environments, which is manifested as reduced brightness and contrast, increased noise, loss of details and color deviation.

[0004] Furthermore, the related low-light image enhancement model based on convolutional neural network aims to reduce the impact of insufficient lighting on image quality by improving image brightness and restoring key details.

[0005] However, most image enhancement algorithms do not implement end-to-end training with detectors or detection networks, but instead train the enhancement and detection algorithms separately, which limits the detection performance. Summary of the invention

[0006] In view of this, the purpose of the present application is to propose an end-to-end low-light image detection method and related equipment based on frequency domain enhancement.

[0007] Based on the above objectives, the present application provides an end-to-end low-light image detection method based on frequency domain enhancement, including:

[0008] Inputting the low-light training image for training into the frequency domain enhanced target detection network to be trained, and using the Fourier enhancement sub-network in the frequency domain enhanced target detection network to perform fast Fourier transform on the low-light training image to obtain a frequency domain feature map;

[0009] Performing complex convolution on the frequency domain feature map, and continuously performing multiple frequency domain processing on the result of the complex convolution to obtain a spectrum feature map, performing dimension reduction on the spectrum feature map to obtain a first frequency domain conversion map, and using a preset activation function to constrain the range of the first frequency domain conversion map to obtain a second frequency domain conversion map;

[0010] Determine the amplitude and phase of the frequency domain feature map, amplify the amplitude of the frequency domain feature map by dividing the amplitude of the frequency domain feature map by the second frequency domain conversion map, and perform inverse fast Fourier transform using the phase and the amplified amplitude to obtain an enhanced image;

[0011] The low-light training image and the enhanced image are input into the target detection subnetwork in the frequency domain enhanced target detection network, the frequency domain enhanced target detection network is end-to-end trained using a preset joint loss function in the target detection subnetwork, and the trained frequency domain enhanced target detection network is used to identify the target body in the low-light image to be detected.

[0012] Further, the low-light training image is subjected to a fast Fourier transform using the Fourier enhancement sub-network in the frequency domain enhanced target detection network to obtain a frequency domain feature map, including:

[0013] The low-light training image is transformed using the fast Fourier transform formula shown below to obtain the frequency domain feature map:

[0014]

[0015] Among them, F(u,v) represents the frequency domain feature map, u and v represent the coordinates of the frequency domain feature map, f(x,y) represents the low-light training image, x and y represent the coordinates of the low-light training image, M represents the width of the low-light training image, N represents the height of the low-light training image, j represents the imaginary unit, and e represents the natural logarithm.

[0016] Among them, the Fourier enhanced sub-network includes complex convolutional layers;

[0017] Further, performing complex convolution on the frequency domain feature map includes:

[0018] The frequency domain feature map is input into the complex convolution layer, and the frequency domain feature map is subjected to synchronization processing of amplitude and phase in the frequency domain using the following formula:

[0019] W*z=(J*α-K*β)+j(K*α+J*β).

[0020] Wherein, W represents the complex filter matrix of the convolution kernel of the complex convolution layer, z represents the complex matrix of the frequency domain feature map, J and K represent the real matrix of W, and α and β represent the real matrix of z;

[0021] Determine the complex convolution result in matrix form using the formula shown below:

[0022]

[0023] Among them, R(W*z) represents the real part of the complex convolution result, and I(W*z) represents the imaginary part of the complex convolution result.

[0024] Among them, the Fourier enhancement sub-network includes multiple frequency domain processing units connected in series, each frequency domain processing unit includes a first residual submodule, a second residual submodule and a third residual submodule, the first residual submodule includes a complex depth-separable convolution layer, the second residual submodule includes a complex array normalization layer, multiple complex convolution layers and a complex depth-separable convolution layer, and the third residual submodule includes a complex activation function, multiple complex convolution layers and a complex depth-separable convolution layer;

[0025] Furthermore, the result of the complex convolution is processed in the frequency domain for multiple times to obtain a spectrum feature map, including:

[0026] Inputting the result of the complex convolution into the first residual submodule of the first frequency domain processing unit, and performing complex depthwise separable convolution to obtain a first output image;

[0027] Inputting the first output image to the second residual submodule of the first frequency domain processing unit, normalizing the first output image using the complex array normalization layer, inputting the normalized result to the complex convolution layer, inputting the complex convolution result to the complex depthwise separable convolution layer, inputting the complex depthwise separable convolution result to another complex convolution layer, adjusting the convolution result of the other complex convolution layer using a preset weight matrix, and combining the adjusted result with the first output image to obtain a second output image;

[0028] Inputting the second output map to the third residual submodule of the first frequency domain processing unit, using the complex activation function, inputting the operation result of the complex activation function to the complex convolution layer, inputting the result of the complex convolution to the complex depthwise separable convolution layer, inputting the result of the complex depthwise separable convolution to another complex convolution layer, adjusting the convolution result of the other complex convolution layer using a preset weight matrix, and combining the adjustment result with the second output map to obtain a third output map;

[0029] The third output graph is input to the next frequency domain processing unit for operation, and the third output graph output by the last frequency domain processing unit is used as the frequency spectrum feature graph.

[0030] Further, determining the amplitude and phase of the frequency domain feature map includes:

[0031] The frequency domain feature map is expressed in complex form as shown below:

[0032] F(u,v)=R(u,v)+jI(u,v).

[0033] Wherein, R(u,v) represents the real part of the frequency domain feature map, and I(u,v) represents the imaginary part of the frequency domain feature map;

[0034] The amplitude of the frequency domain feature map is expressed as:

[0035]

[0036] Wherein, A(u,v) represents the amplitude;

[0037] The phase of the frequency domain feature map is expressed as:

[0038]

[0039] Wherein, P(u,v) represents the phase.

[0040] Further, the amplified amplitude and the phase are used to perform an inverse fast Fourier transform to obtain an enhanced image, including:

[0041] According to the formula shown below, the real part and the imaginary part of the frequency domain characteristic graph after the amplitude is amplified are determined using the amplified amplitude and the phase:

[0042]

[0043] Among them, A′(u,v) represents the amplitude after amplification, R′(v,u) represents the real part of the frequency domain feature map after amplification, and I′(u,v) represents the imaginary part of the frequency domain feature map after amplification;

[0044] The real part and the imaginary part of the frequency domain characteristic graph after the amplitude is amplified are used to determine the frequency domain characteristic graph after the amplitude is amplified as shown below:

[0045] F′(u,v)=R′(u,v)+jI′(u,v).

[0046] Among them, F′(u,v) represents the frequency domain feature map after amplitude amplification;

[0047] The enhanced image is obtained by inversely transforming the frequency domain feature map after amplitude amplification using the inverse fast Fourier transform formula shown below:

[0048]

[0049] Wherein, f′(u,v) represents the enhanced image.

[0050] The joint loss function includes an enhancement loss function and a detection loss function, and the enhancement loss function includes an amplitude difference loss function and a phase similarity loss function;

[0051] Furthermore, the frequency domain enhanced target detection network is end-to-end trained using a preset joint loss function in the target detection subnetwork, including:

[0052] The amplitude difference loss function is constructed as follows:

[0053] L amp =D icd (A input (u,v),A enhance (u,v)).

[0054] Among them, L amp represents the amplitude difference loss, A input (u,v) represents the amplitude of the low-light training image used for training, A enhance (u, v) represents the amplitude of the enhanced image, D icd (A input (u,v),A enhance (u,v)) represents the magnitude difference of Chebyshev distance;

[0055] Construct the phase similarity loss function as shown below:

[0056] L phase =1-SSIM(P input (u,v),P enhance (u,v)).

[0057] Among them, SSIM(P input (u,v),P enhance (u,v)) represents the phase structure similarity between the low-light training image and the enhanced image in the frequency domain.

[0058] Based on the same inventive concept, the present application also provides an end-to-end low-light image detection device based on frequency domain enhancement, comprising: a transformation module, a frequency domain processing module, an enhancement module and a training module;

[0059] The transformation module is configured to input the low-light training image for training into the frequency domain enhanced target detection network to be trained, and use the Fourier enhancement sub-network in the frequency domain enhanced target detection network to perform fast Fourier transformation on the low-light training image to obtain a frequency domain feature map;

[0060] The frequency domain processing module is configured to perform complex convolution on the frequency domain feature map, and continuously perform multiple frequency domain processing on the result of the complex convolution to obtain a spectrum feature map, perform dimension reduction on the spectrum feature map to obtain a first frequency domain conversion map, and use a preset activation function to constrain the range of the first frequency domain conversion map to obtain a second frequency domain conversion map;

[0061] The enhancement module is configured to determine the amplitude and phase of the frequency domain feature map, amplify the amplitude of the frequency domain feature map by dividing the amplitude of the frequency domain feature map by the second frequency domain conversion map, and perform an inverse fast Fourier transform using the phase and the amplified amplitude to obtain an enhanced image;

[0062] The training module is configured to input the low-light training image and the enhanced image into the target detection subnetwork in the frequency domain enhanced target detection network, perform end-to-end training on the frequency domain enhanced target detection network using a preset joint loss function in the target detection subnetwork, and use the trained frequency domain enhanced target detection network to identify the target body in the low-light image to be detected.

[0063] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and run on the processor. When the processor executes the program, it implements the end-to-end low-light image detection method based on frequency domain enhancement as described in any one of the above.

[0064] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the end-to-end low-light image detection method based on frequency domain enhancement as described above.

[0065] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the end-to-end low-light image detection method based on frequency domain enhancement as described in any of the above items.

[0066] From the above, it can be seen that the end-to-end low-light image detection method and related equipment based on frequency domain enhancement provided by the present application are based on the fast Fourier transform of the low-light image, so as to convert the low-light image from the spatial domain to the frequency domain to obtain a frequency domain feature map, and obtain a spectrum feature map by multiple frequency domain processing of the frequency domain feature map. After the dimensionality reduction and range constraint of the spectrum feature map, the frequency domain conversion map is obtained. At the same time, after determining the amplitude and phase of the frequency domain feature map, the spectrum conversion map is used to amplify the amplitude, and the amplified amplitude and phase are used to perform a fast Fourier inverse transform to obtain an enhanced image.

[0067] Furthermore, the frequency domain enhanced target detection network is trained end-to-end by setting a joint loss function. The influence of amplitude and phase on the image is comprehensively considered, and amplitude difference loss and phase similarity loss are added to the joint loss function for joint training, so that the Fourier enhancement sub-network and the target detection sub-network can be trained simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 This is a flow chart of an end-to-end low-light image detection method based on frequency domain enhancement according to an embodiment of the present application;

[0070] Figure 2 This is a structural diagram of a frequency domain enhanced target detection network according to an embodiment of the present application;

[0071] Figure 3 This is a structural diagram of a frequency domain processing unit according to an embodiment of the present application;

[0072] Figure 4 A schematic diagram of a weight matrix of a frequency domain processing unit in an embodiment of the present application;

[0073] Figure 5 This is a schematic diagram of the structure of an end-to-end low-light image detection device based on frequency domain enhancement according to an embodiment of the present application;

[0074] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0076] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0077] As described in the background technology section, the related low-light image detection technology is still difficult to meet the needs of people in actual work.

[0078] In the process of implementing this application, the applicant discovered that the main problems with the relevant low-light image detection methods are that low-light image target detection algorithms based on deep learning generally have the problems of low accuracy, high computational complexity, and insufficient model generalization ability.

[0079] Specifically, many target detection algorithms perform well on images under normal lighting conditions, but cannot achieve ideal results in low-light or dark-light environments, such as nighttime autonomous driving, face detection, security monitoring and other applications. This is because the image quality is significantly degraded in dark-light environments, which is manifested as reduced brightness and contrast, increased noise, loss of details and color deviation.

[0080] Furthermore, the related low-light image enhancement model based on convolutional neural network aims to reduce the impact of insufficient lighting on image quality by improving image brightness and restoring key details.

[0081] However, most image enhancement algorithms do not implement end-to-end training with detectors or detection networks, but instead train the enhancement and detection algorithms separately, which limits the detection performance.

[0082] Based on this, one or more embodiments in the present application provide an end-to-end low-illumination image detection method based on frequency domain enhancement.

[0083] In an embodiment of the present application, a frequency domain enhanced target detection network for low-illumination image target detection includes two sub-networks, a Fourier enhancement sub-network and a target detection sub-network, which are connected in series.

[0084] Among them, the Fourier enhancement subnetwork is used to enhance the low-light image. The low-light image is first input into the Fourier enhancement subnetwork to obtain an enhanced image with improved brightness. When the enhancement network is input into the serial target detection subnetwork, the target detection subnetwork is used to identify the target body in the enhanced image.

[0085] The target detection subnetwork may be, for example, a YOLO network, YOLOv3 (the third version of the YOLO network), or YOLOv8 (the eighth version of the YOLO network), or other versions of the YOLO network. The YOLO network is a target detection network that can simultaneously predict multiple physical categories and positions in an image through a single forward pass.

[0086] In this embodiment, the frequency domain enhanced target detection network composed of the Fourier enhancement sub-network and the target detection sub-network is trained end-to-end as a whole, and the trained frequency domain enhanced target detection network is used to identify the target body in the low-light image to be detected.

[0087] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0088] refer to Figure 1 , an end-to-end low-light image detection method based on frequency domain enhancement according to an embodiment of the present application comprises the following steps:

[0089] Step S101: input a low-light training image for training into a frequency domain enhanced target detection network to be trained, and use a Fourier enhancement sub-network in the frequency domain enhanced target detection network to perform a fast Fourier transform on the low-light training image to obtain a frequency domain feature map.

[0090] In an embodiment of the present application, a frequency domain enhanced target detection network is trained using pre-prepared low-light images for training. After the low-light images for training are input into the frequency domain enhanced target detection network, a fast Fourier transform is first performed on the low-light images for training to obtain a frequency domain feature map.

[0091] Specifically, Figure 2 A specific example of a frequency domain enhanced target detection network is shown. When a low-light image used for training is input into the frequency domain enhanced target detection network, it is first input into a Fourier enhancement sub-network, and the Fourier enhancement sub-network performs a fast Fourier transform on the low-light image used for training, thereby transferring the low-light image from the spatial domain to the frequency domain.

[0092] Among them, the fast Fourier transform formula shown below can be used to transform the low-light image used for training to obtain the frequency domain feature map:

[0093]

[0094] Among them, F(u,v) represents the frequency domain feature map, u and v represent the coordinates of the frequency domain feature map, f(x,y) represents the low-illumination training image, x and y represent the coordinates of the low-illumination training image, M represents the width of the low-illumination training image, N represents the height of the low-illumination training image, j represents the imaginary unit, and e represents the natural logarithm.

[0095] Based on this, Fourier frequency domain information can be obtained through fast Fourier transform, that is, the frequency domain feature map containing the global context can be extracted. Especially in the scene of low-light images, the importance of Fourier frequency domain information is more prominent, which can effectively enhance the feature extraction and representation performance of the Fourier enhancement sub-network for low-light images.

[0096] In this implementation, in the frequency domain features of the frequency domain feature map, the real part represents the amplitude and reflects the brightness and contrast, and the imaginary part represents the phase and reflects the structure, details and noise information. The amplitude component in the frequency domain is expanded through the complex convolution layer and the frequency domain processing unit in the following steps, thereby effectively improving the overall brightness and contrast of the low-light image and optimizing the image visual effect.

[0097] Step S102, perform complex convolution on the frequency domain feature map, and continuously perform multiple frequency domain processing on the result of the complex convolution to obtain a spectrum feature map, perform dimensionality reduction on the spectrum feature map to obtain a first frequency domain conversion map, use a preset activation function to constrain the range of the first frequency domain conversion map, and obtain a second frequency domain conversion map.

[0098] In an embodiment of the present application, based on the frequency domain feature map determined in the aforementioned steps, a spectrum feature map is obtained by performing complex convolution and multiple continuous frequency domain processing on the frequency domain feature map, and a conversion map of the spectrum feature map is obtained through dimensionality reduction and activation function constraints.

[0099] Specifically, in Figure 2 In the specific example shown, a complex convolution layer, three series-connected frequency domain processing units, a convolution layer and a Sigmoid function (activation function) are sequentially arranged in the Fourier enhancement subnetwork. When the frequency spectrum feature map obtained in the above steps is input into the Fourier enhancement subnetwork, it is first input into the complex convolution layer, and the complex convolution layer is used to perform a complex convolution operation on the frequency domain feature map.

[0100] In this embodiment, since the traditional convolutional network can only process real data in the spatial domain but cannot process complex data in the frequency domain, the complex convolution layer can be used to synchronously process the amplitude and phase of the frequency domain feature map in the frequency domain.

[0101] Among them, the operation of complex convolution is expressed as follows:

[0102] W*z=(J*α-K*β)+j(K*α+J*β).

[0103] Among them, j is an imaginary unit, W represents the complex filter matrix of the convolution kernel of the complex convolution layer, and is specifically expressed in complex form as J+jK, z represents the complex matrix of the frequency domain feature map, and is specifically expressed in complex form as α+jβ, J and K represent the real matrix of W, and α and β represent the real matrix of z.

[0104] Based on this, the complex convolution result in matrix form is determined using the formula shown below:

[0105]

[0106] Among them, R(W*z) represents the real part of the complex convolution result, and I(W*z) represents the imaginary part of the complex convolution result.

[0107] In this embodiment, the operation of complex convolution is represented as ComplexConv(z), and the result of complex convolution is represented as I f = ComplexConv(z).

[0108] Based on this, Figure 2 As shown, the complex convolution result is input into three frequency domain processing units connected in series, and after three times of frequency domain processing, a spectrum feature map is obtained.

[0109] In this embodiment, the three frequency domain processing operations are represented as follows: fout =FPB(FPB(FPB(I f ))), where FPB represents frequency domain processing unit, I out It represents the frequency spectrum feature map output by the last frequency domain processing unit after passing through three frequency domain processing units.

[0110] Specifically, the above Figure 2 The complex convolution result of the complex convolution layer is input to the first frequency domain processing unit, and the first frequency domain processing unit performs frequency domain processing to obtain a first frequency domain processing graph.

[0111] Furthermore, the first frequency domain processing unit inputs the first frequency domain processing graph to the next frequency domain processing unit, and the next frequency domain processing unit performs frequency domain processing to obtain a second frequency domain processing graph.

[0112] Furthermore, the second frequency domain processing unit inputs the obtained second frequency domain processing graph into the third frequency domain processing unit, that is, the last frequency domain processing unit, and after the last frequency domain processing unit performs frequency domain processing, a third frequency domain processing graph is obtained. In this embodiment, the third frequency domain processing graph output by the last frequency domain processing unit is used as a spectrum feature graph.

[0113] In a specific embodiment, each frequency domain processing unit has the same structure. Figure 3 The specific structure of each frequency domain processing unit is shown. It can be seen that in each frequency domain processing unit, three residual submodules are connected in series, namely, the first residual submodule, the second residual submodule and the third residual submodule.

[0114] In this embodiment, if Figure 3 As shown, a complex depth-separable convolution layer (DWCConv) with a convolution kernel of 3x3 is provided in the first residual submodule.

[0115] Furthermore, if Figure 3 As shown, the second residual submodule is sequentially connected in series with a complex array normalization layer (ComplexGN), a complex convolution layer (ComplexConv) with a convolution kernel of 1ⅹ1, a complex depth-separable convolution layer (DWCConv) with a convolution kernel of 5ⅹ5, another complex convolution layer (ComplexConv) with a convolution kernel of 1ⅹ1, and a preset weight matrix.

[0116] Furthermore, if Figure 3 As shown, the third residual submodule is sequentially connected in series with a complex activation function (ComplexELU), a complex convolution layer (ComplexConv) with a convolution kernel of 1×1, a complex depth-separable convolution layer (DWCConv) with a convolution kernel of 3×3, another complex convolution layer (ComplexConv) with a convolution kernel of 1×1, and a preset weight matrix.

[0117] Based on this, Figure 2 When the complex convolution result of the complex convolution layer is input to the first frequency domain processing unit, the complex depthwise separable convolution layer in the first residual submodule performs complex depthwise separable convolution on the result of the complex convolution to obtain a first output image.

[0118] Among them, the complex depthwise separable convolution layer combines the advantages of depthwise convolution and pointwise convolution, and processes complex data. In the complex depthwise convolution stage, each convolution kernel focuses on processing a complex channel of the input feature map, including the real part and the imaginary part.

[0119] Furthermore, in the complex point convolution stage, the input feature map, that is, the complex convolution result, is integrated in the direction of the complex channel, and the complex weight matrix is ​​used to combine the features of different channels to extract deeper complex information.

[0120] In some embodiments, when the scales of the input feature map, the output feature map, and the convolution kernel are H×H×C, H×H×D, and F×F, respectively, the computational expressions of the complex convolution operation and the complex depth-separable convolution operation are as follows:

[0121]

[0122] Among them, J ComplexConv represents the computational complexity of the complex convolution operation, J DWCConv Represents the computational amount of complex depthwise separable convolution. Through the above computational amount expression, it can be found that the complex depthwise separable convolution operation can effectively reduce the computational amount compared with the complex convolution operation, thereby improving the inference speed.

[0123] In this embodiment, after the first residual submodule obtains the first output image, the first output image is input to the second residual submodule, such as Figure 3 As shown, the complex group normalization layer in the second residual submodule performs group normalization on the first output image, and obtains the first subimage of the second residual submodule.

[0124] Among them, based on group normalization, the frequency characteristics of the real part and the imaginary part are unified. Compared with batch normalization, group normalization can reduce the impact of batch normalization on the performance of small batch images. By grouping and normalizing similar texture and brightness features, the brightness can be effectively stabilized and the brightness difference between image areas can be reduced, thereby improving image clarity.

[0125] Furthermore, the complex array normalization layer outputs the first sub-image to a complex convolution layer with a convolution kernel of 1x1, and the complex convolution layer performs a complex convolution operation on the first sub-image, thereby extracting frequency domain features and obtaining the second sub-image of the second residual sub-module.

[0126] Furthermore, the complex convolution layer outputs the second sub-image to a 5x5 complex depthwise separable convolution layer, and the complex depthwise separable convolution layer performs a complex depthwise separable convolution on the second sub-image to achieve complex feature learning at low computational cost, and obtain the third sub-image of the second residual sub-module.

[0127] Furthermore, the complex depthwise separable convolutional layer outputs the third sub-image to another 1x1 complex convolutional layer, and the complex convolutional layer performs another complex convolution operation on the third sub-image to obtain the fourth sub-image of the second residual sub-module.

[0128] Furthermore, the weight matrix pre-set in the second residual submodule is used to perform a dot multiplication with the fourth subgraph, so as to adjust the output of each layer, optimize the feature distribution, and achieve inter-layer feature balance.

[0129] Among them, Figure 4 As shown, the weight matrix can be a diagonal matrix containing 1 parameter. Figure 4 In the example of , λl and λd are , and the elements in the diagonal positions are parameters.

[0130] Based on this, layer scaling can be achieved by adjusting the parameters of the weight matrix, which not only equips each channel with appropriate parameters and enhances adaptive capabilities, but also improves the dynamics of training.

[0131] Furthermore, if Figure 3 As shown, the result of the dot product of the weight matrix and the fourth sub-image is combined with the first output image input to the second residual sub-module to obtain the second output image, and the second output image is output to the third residual sub-module.

[0132] In this embodiment, the complex activation function in the third residual submodule processes the second output image according to the following calculation formula to obtain the first subimage of the third residual submodule:

[0133] ComplexELU(z)=ReLU(R(z))+jReLU(I(z)).

[0134] Among them, ComplexELU(z) represents the complex activation function, ReLU(R(z)) represents the real part of the complex activation function, and ReLU(I(z)) represents the imaginary part of the complex activation function.

[0135] Based on this, in the third residual submodule, a complex activation function is used to replace the complex array normalization layer in the second residual submodule, which can not only retain high-frequency information, but also enhance low-frequency features through nonlinear transformation, thereby improving the model's expression ability.

[0136] Furthermore, the obtained first sub-image is output to a complex convolution layer with a convolution kernel of 1x1, and the complex convolution layer performs a complex convolution operation on the first sub-image, thereby extracting frequency domain features and obtaining the second sub-image of the third residual sub-module.

[0137] Furthermore, the complex convolution layer outputs the second sub-image to a 3x3 complex depthwise separable convolution layer, and the complex depthwise separable convolution layer performs a complex depthwise separable convolution on the second sub-image to obtain a third sub-image of the second residual sub-module.

[0138] Among them, the third residual submodule uses a 3×3 complex depth-separable convolutional layer, which is different from the 5×5 complex depth-separable convolutional layer in the second residual submodule, and can extract multi-scale feature information.

[0139] Furthermore, the complex depthwise separable convolutional layer outputs the third sub-image to another 1x1 complex convolutional layer, and the complex convolutional layer performs another complex convolution operation on the third sub-image to obtain the fourth sub-image of the third residual sub-module.

[0140] Furthermore, the same weight matrix as that of the second residual submodule is pre-set in the third residual submodule, and the weight matrix is ​​used to perform dot multiplication with the fourth subgraph, so as to adjust the output of each layer.

[0141] Furthermore, if Figure 3 As shown, the result of the dot product of the weight matrix and the fourth sub-graph is combined with the second output graph input to the third residual sub-module to obtain a third output graph, and the third output graph is output to the first residual sub-module of the next frequency domain processing unit for operation.

[0142] In this embodiment, the second and third frequency domain processing units have the same structure as the first frequency domain processing unit and perform the same operation.

[0143] in, Figure 2 The third output image of the third residual submodule in the second frequency domain processing is the second frequency domain processing image; the third output image of the third residual submodule in the third frequency domain processing unit, that is, the last frequency domain processing unit, is the third frequency domain processing image; accordingly, the third output image of the last frequency domain processing unit is used as the spectrum feature image.

[0144] In this embodiment, if Figure 2 As shown, the frequency domain feature map obtained after three series-connected frequency domain processing units is input into a 1x1 convolution layer, and the convolution layer is used to perform dimensionality reduction processing on the frequency domain feature map to obtain a first frequency domain conversion map.

[0145] Furthermore, if Figure 2 As shown, a Sigmoid function is provided in the Fourier enhancement subnetwork, and the first frequency domain conversion graph is input into the Sigmoid function for operation, thereby constraining the range of the first frequency domain conversion graph to be within (0,1), and obtaining the second frequency domain conversion graph.

[0146] In this embodiment, the spectrum characteristic graph I out The operations for dimensionality reduction and range constraints are expressed as follows:

[0147] T = Sigmoid(Conv(I fout )).

[0148] Wherein, T represents the second frequency domain conversion diagram.

[0149] Step S103, determine the amplitude and phase of the frequency domain feature map, amplify the amplitude of the frequency domain feature map by dividing the amplitude of the frequency domain feature map by the second frequency domain conversion map, perform inverse fast Fourier transform using the phase and the amplified amplitude, and obtain an enhanced image.

[0150] In the embodiments of the present application, Figure 2As shown, based on the frequency domain feature map obtained after fast Fourier transform in the aforementioned steps, the amplitude and phase of the frequency domain feature map can be determined, and after amplifying the amplitude, the amplified amplitude and phase are used to perform inverse fast Fourier transform to obtain an enhanced image.

[0151] Specifically, the frequency domain feature map obtained in the previous steps is expressed in the complex form as shown below:

[0152] F(u,v)=R(u,v)+jI(u,v).

[0153] Among them, R(u,v) represents the real part of the frequency domain feature map, and I(u,v) represents the imaginary part of the frequency domain feature map.

[0154] Based on this, the amplitude and phase of the frequency domain feature graph are determined using the real part and imaginary part of the frequency domain feature graph.

[0155] Specifically, the amplitude of the frequency domain feature map is expressed as follows:

[0156]

[0157] Among them, A(u,v) represents the amplitude of the frequency domain feature map.

[0158] Furthermore, the phase of the frequency domain feature map is expressed as follows:

[0159]

[0160] Among them, P(u,v) represents the phase of the frequency domain feature map.

[0161] Based on this, Figure 2 As shown, by dividing the amplitude of the determined frequency domain feature map by the second frequency domain conversion map T determined above, the amplitude of the frequency domain feature map can be amplified, thereby further improving the brightness of the low-light image, making the originally darker areas brighter to reveal more image details.

[0162] Among them, the amplified amplitude is represented as A′(u,v),

[0163] The amplified amplitude and phase are used to construct the formula shown below to determine the real and imaginary parts of the frequency domain characteristic graph after amplitude amplification:

[0164]

[0165] Among them, R′(u,v) represents the real part of the frequency domain feature map after the amplitude is amplified, and I′(u,v) represents the imaginary part of the frequency domain feature map after the amplitude is amplified.

[0166] Based on this, the frequency domain feature map after amplification can be expressed as follows:

[0167] F′(u,v)=R′(u,v)+jI′(u,v).

[0168] Among them, F′(u,v) represents the frequency domain feature map after amplitude amplification.

[0169] Furthermore, if Figure 2 As shown, the frequency domain feature map F′(u,v) after the amplification is subjected to the inverse fast Fourier transform as shown below, and the enhanced image is obtained after the inverse fast Fourier transform:

[0170]

[0171] Where f′(u,v) represents the enhanced image.

[0172] Step S104: input the low-light training image and the enhanced image into the target detection subnetwork in the frequency domain enhanced target detection network, perform end-to-end training on the frequency domain enhanced target detection network using a preset joint loss function in the target detection subnetwork, and use the trained frequency domain enhanced target detection network to identify the target body in the low-light image to be detected.

[0173] In an embodiment of the present application, after the enhanced image is determined using the Fourier enhancement sub-network, the enhanced image and the low-light image used for training are input into the target detection sub-network, and the target detection sub-network is trained end-to-end based on the joint loss function set in the target detection sub-network.

[0174] Specifically, Figure 2 In the specific example shown, the target detection subnetwork is YOLOv3, and a joint loss function is set therein to replace the original loss function of YOLOv3.

[0175] Among them, the joint loss function includes the enhancement loss function and the detection loss function, and is specifically expressed as follows:

[0176] L total =λ·L enhancement +(1-λ)L detection .

[0177] Among them, L total represents the joint loss, L enhancemen represents the enhancement loss, L detection represents the detection loss function, λ represents a preset weight factor, the weight factor may be, for example, 0.4, and the detection loss function may be the original loss in YOLOv3.

[0178] In this embodiment, in order to solve the problems of insufficient brightness, low contrast and detail loss in target detection of low-illuminance images, the amplitude difference loss and the phase similarity loss are combined to construct an enhancement loss function.

[0179] It can be seen that the goal of the enhanced loss function constructed in this way is to optimize and maintain the amplitude and phase of Fourier information, thereby improving the target detection performance.

[0180] Specifically, the enhanced loss function is constructed as follows:

[0181] L enhancement =L amp +L phase .

[0182] Among them, L amp Represents the amplitude difference loss, L phase represents the phase similarity loss.

[0183] In this embodiment, since the amplitude has a significant impact on the intensity and brightness of the pixel, the amplitude difference loss is used to constrain the amplitude difference between the enhanced image and the low-light image. By emphasizing the amplitude change, not only the brightness level of the enhanced image can be effectively regulated, but also the brightness of the enhanced image can be effectively controlled, thereby achieving the effect of maximizing the amplitude difference.

[0184] Among them, in the amplitude difference loss, the amplitude difference using the Chebyshev distance can effectively reflect the absolute value of the amplitude difference, thereby more clearly highlighting the amplitude change between the enhanced image and the low-illumination image.

[0185] Based on this, the amplitude difference loss function is specifically expressed as follows:

[0186] L amp =D icd (A input (u,v),A enhance (u,v)).

[0187] Among them, A input (u,v) represents the amplitude of the low-light training image used for training, A enhance (u,v) represents the amplitude of the enhanced image, D icd (A input (u,v),A enhance (u,v)) represents the magnitude difference of the Chebyshev distance.

[0188] Furthermore, in the amplitude difference function, the amplitude difference of the Chebyshev distance is determined according to the following formula:

[0189]

[0190] Wherein, n represents the rank of the image matrix. Since the rank of the image matrix of the low-light image input in this embodiment is the same as the rank of the image matrix of the enhanced image, n can be the rank of the image matrix of the low-light image or the rank of the image matrix of the enhanced image. cd (A input (u,v),A enhance (u,v)) is the Chebyshev distance between the enhanced image and the low-light image, represents the average Chebyshev distance between the low-light image and the enhanced image, λ max represents the maximum eigenvalue of the enhanced image matrix, and its function is to reflect the difference in Chebyshev distance between images under different lighting conditions through amplification. k represents a hyperparameter, and its value can be, for example, 8.

[0191] Furthermore, the Chebyshev distance can be determined according to the following formula:

[0192] D cd (A input (u,v),A enhance (u,v))=max (u,v) |A input (u,v)-A enhance (u,v))|.

[0193] Among them, max (u,v) represents the maximum difference in the frequency domain, and |·| represents the modulus of the complex number.

[0194] In this embodiment, based on the importance of phase information in determining image spatial structure and texture features, maintaining image phase similarity can help restore detailed low-light image details and prevent interference from artifacts and noise, thereby improving target detection accuracy.

[0195] Based on this, the phase similarity loss function aimed at reducing the phase distribution difference between images can effectively constrain the phase consistency between images.

[0196] Specifically, the phase similarity loss function is constructed as follows:

[0197] L phase =1-SSIM(P input (u,v),P enhance (u,v)).

[0198] Among them, SSIM(P input (u,v),P enhance (u,v)) represents the phase structure similarity between the low-light image and the enhanced image in the frequency domain.

[0199] Furthermore, the phase structure similarity can be specifically calculated according to the following formula:

[0200]

[0201] Among them, P input (u,v) represents the phase of the low-light image, P enhance (u,v) represents the phase of the enhanced image, μ I Represents the phase mean of the low-light image, μ J represents the phase mean of the enhanced image, σ I Represents the phase standard deviation of the low-light image, σ J represents the phase standard deviation of the enhanced image, σ IJ represents the phase covariance of the illuminance image and the enhanced image, and c1 and c2 are both constants.

[0202] According to this, 1-SSIM(P input (u,v),P enhance (u,v)) can represent the minimization of the phase structure similarity ratio.

[0203] In this implementation, based on the above-mentioned joint loss function, when the frequency domain enhanced target detection network is trained end-to-end as a whole, the enhanced image and the low-light image are used to determine the joint loss in the process of using the target detection subnetwork to recognize the enhanced image. When the joint loss is less than or equal to the preset loss threshold, the frequency domain enhanced target detection network training is considered to be completed.

[0204] Based on this, the trained frequency domain enhanced target detection network can be used to identify the target in the low-light image to be detected.

[0205] It can be seen that the end-to-end low-light image detection method based on frequency domain enhancement in the embodiment of the present application is based on the fast Fourier transform of the low-light image, so as to convert the low-light image from the spatial domain to the frequency domain to obtain a frequency domain feature map, and obtain a spectrum feature map through multiple frequency domain processing of the frequency domain feature map. After the dimensionality reduction and range constraints of the spectrum feature map, the frequency domain conversion map is obtained. At the same time, after determining the amplitude and phase of the frequency domain feature map, the spectrum conversion map is used to amplify the amplitude, and the amplified amplitude and phase are used to perform inverse fast Fourier transform to obtain an enhanced image.

[0206] Furthermore, the frequency domain enhanced target detection network is trained end-to-end by setting a joint loss function. The influence of amplitude and phase on the image is comprehensively considered, and amplitude difference loss and phase similarity loss are added to the joint loss function for joint training, so that the Fourier enhancement sub-network and the target detection sub-network can be trained simultaneously.

[0207] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0208] It should be noted that some embodiments of the present application are described above. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the above-described embodiments and still achieve the desired result. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0209] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application further provides an end-to-end low-illumination image detection device based on frequency domain enhancement.

[0210] refer to Figure 5 The end-to-end low-light image detection device based on frequency domain enhancement includes: a transformation module 501, a frequency domain processing module 502, an enhancement module 503 and a training module 504;

[0211] The transformation module 501 is configured to input the low-light training image for training into the frequency domain enhanced target detection network to be trained, and use the Fourier enhancement sub-network in the frequency domain enhanced target detection network to perform fast Fourier transformation on the low-light training image to obtain a frequency domain feature map;

[0212] The frequency domain processing module 502 is configured to perform complex convolution on the frequency domain feature map, and continuously perform multiple frequency domain processing on the result of the complex convolution to obtain a spectrum feature map, perform dimension reduction on the spectrum feature map to obtain a first frequency domain conversion map, and use a preset activation function to constrain the range of the first frequency domain conversion map to obtain a second frequency domain conversion map;

[0213] The enhancement module 503 is configured to determine the amplitude and phase of the frequency domain feature map, amplify the amplitude of the frequency domain feature map by dividing the amplitude of the frequency domain feature map by the second frequency domain conversion map, and perform an inverse fast Fourier transform using the phase and the amplified amplitude to obtain an enhanced image;

[0214] The training module 504 is configured to input the low-light training image and the enhanced image into the target detection subnetwork in the frequency domain enhanced target detection network, perform end-to-end training on the frequency domain enhanced target detection network using a preset joint loss function in the target detection subnetwork, and use the trained frequency domain enhanced target detection network to identify the target body in the low-light image to be detected.

[0215] For the convenience of description, the above devices are described in terms of functions and are divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0216] The device of the above embodiment is used to implement the corresponding end-to-end low-light image detection method based on frequency domain enhancement in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0217] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the end-to-end low-light image detection method based on frequency domain enhancement as described in any of the above embodiments is implemented.

[0218] Figure 6 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0219] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0220] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0221] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0222] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0223] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0224] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0225] The device of the above embodiment is used to implement the corresponding end-to-end low-light image detection method based on frequency domain enhancement in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0226] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the end-to-end low-light image detection method based on frequency domain enhancement as described in any of the above embodiments.

[0227] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0228] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the end-to-end low-light image detection method based on frequency domain enhancement as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0229] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the end-to-end low-light image detection method based on frequency domain enhancement as described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.

[0230] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0231] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0232] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0233] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. An end-to-end low-light image detection method based on frequency domain enhancement, characterized in that: include: Inputting the low-light training image for training into the frequency domain enhanced target detection network to be trained, and using the Fourier enhancement sub-network in the frequency domain enhanced target detection network to perform fast Fourier transform on the low-light training image to obtain a frequency domain feature map; Performing complex convolution on the frequency domain feature map, and continuously performing multiple frequency domain processing on the result of the complex convolution to obtain a spectrum feature map, performing dimension reduction on the spectrum feature map to obtain a first frequency domain conversion map, and using a preset activation function to constrain the range of the first frequency domain conversion map to obtain a second frequency domain conversion map; Determine the amplitude and phase of the frequency domain feature map, amplify the amplitude of the frequency domain feature map by dividing the amplitude of the frequency domain feature map by the second frequency domain conversion map, and perform inverse fast Fourier transform using the phase and the amplified amplitude to obtain an enhanced image; The low-light training image and the enhanced image are input into the target detection subnetwork in the frequency domain enhanced target detection network, the frequency domain enhanced target detection network is end-to-end trained using a preset joint loss function in the target detection subnetwork, and the trained frequency domain enhanced target detection network is used to identify the target body in the low-light image to be detected.

2. The method according to claim 1, characterized in that The method of using the Fourier enhancement sub-network in the frequency domain enhanced target detection network to perform fast Fourier transform on the low illumination training image to obtain a frequency domain feature map includes: The low-light training image is transformed using the fast Fourier transform formula shown below to obtain the frequency domain feature map: Among them, F(u,v) represents the frequency domain feature map, u and v represent the coordinates of the frequency domain feature map, f(x,y) represents the low-light training image, x and y represent the coordinates of the low-light training image, M represents the width of the low-light training image, N represents the height of the low-light training image, j represents the imaginary unit, and e represents the natural logarithm.

3. The method according to claim 1, characterized in that The Fourier enhancer network includes a complex convolution layer; The performing complex convolution on the frequency domain feature map comprises: The frequency domain feature map is input into the complex convolution layer, and the frequency domain feature map is subjected to synchronization processing of amplitude and phase in the frequency domain using the following formula: W*z=(J*α-K*β)+j(K*α+J*β); Wherein, W represents the complex filter matrix of the convolution kernel of the complex convolution layer, z represents the complex matrix of the frequency domain feature map, J and K represent the real matrix of W, and α and β represent the real matrix of z; Determine the complex convolution result in matrix form using the formula shown below: Among them, R(W*z) represents the real part of the complex convolution result, and I(W*z) represents the imaginary part of the complex convolution result.

4. The method according to claim 1, characterized in that: The Fourier enhancement sub-network includes a plurality of frequency domain processing units connected in series, each frequency domain processing unit includes a first residual sub-module, a second residual sub-module and a third residual sub-module, the first residual sub-module includes a complex depth-separable convolution layer, the second residual sub-module includes a complex array normalization layer, a plurality of complex convolution layers and a complex depth-separable convolution layer, and the third residual sub-module includes a complex activation function, a plurality of complex convolution layers and a complex depth-separable convolution layer; The result of the complex convolution is processed in the frequency domain for multiple times continuously to obtain a frequency spectrum feature map, including: Inputting the result of the complex convolution into the first residual submodule of the first frequency domain processing unit, and performing complex depthwise separable convolution to obtain a first output image; Inputting the first output image to the second residual submodule of the first frequency domain processing unit, normalizing the first output image using the complex array normalization layer, inputting the normalized result to the complex convolution layer, inputting the complex convolution result to the complex depthwise separable convolution layer, inputting the complex depthwise separable convolution result to another complex convolution layer, adjusting the convolution result of the other complex convolution layer using a preset weight matrix, and combining the adjusted result with the first output image to obtain a second output image; Inputting the second output map to the third residual submodule of the first frequency domain processing unit, using the complex activation function, inputting the operation result of the complex activation function to the complex convolution layer, inputting the result of the complex convolution to the complex depthwise separable convolution layer, inputting the result of the complex depthwise separable convolution to another complex convolution layer, adjusting the convolution result of the other complex convolution layer using a preset weight matrix, and combining the adjustment result with the second output map to obtain a third output map; The third output graph is input to the next frequency domain processing unit for operation, and the third output graph output by the last frequency domain processing unit is used as the frequency spectrum feature graph.

5. The method according to claim 1, characterized in that Determining the amplitude and phase of the frequency domain feature graph includes: The frequency domain feature map is expressed in complex form as shown below: F(u,v)=R(u,v)+jI(u,v); Wherein, R(u,v) represents the real part of the frequency domain feature map, and I(u,v) represents the imaginary part of the frequency domain feature map; The amplitude of the frequency domain feature map is expressed as: Wherein, A(u,v) represents the amplitude; The phase of the frequency domain feature map is expressed as: Wherein, P(u,v) represents the phase.

6. The method according to claim 1, characterized in that The method of performing inverse fast Fourier transform using the amplified amplitude and the phase to obtain an enhanced image includes: According to the formula shown below, the real part and the imaginary part of the frequency domain characteristic graph after the amplitude is amplified are determined using the amplified amplitude and the phase: Among them, A′(u,v) represents the amplitude after amplification, R′(u,v) represents the real part of the frequency domain feature map after amplification, and I′(u,v) represents the imaginary part of the frequency domain feature map after amplification; The real part and the imaginary part of the frequency domain characteristic graph after the amplitude is amplified are used to determine the frequency domain characteristic graph after the amplitude is amplified as shown below: F′(u,v)=R′(u,v)+jI′(u,v); Among them, F′(u,v) represents the frequency domain feature map after amplitude amplification; The enhanced image is obtained by inversely transforming the frequency domain feature map after amplitude amplification using the inverse fast Fourier transform formula shown below: Wherein, f′(u,v) represents the enhanced image.

7. The method according to claim 1, characterized in that The joint loss function includes an enhancement loss function and a detection loss function, and the enhancement loss function includes an amplitude difference loss function and a phase similarity loss function; The method of performing end-to-end training on the frequency domain enhanced target detection network by using a joint loss function preset in the target detection subnetwork includes: The amplitude difference loss function is constructed as follows: L amp =D icd (A input (u,v),A enhance (u,v)); Among them, L amp represents the amplitude difference loss, A input (u,v) represents the amplitude of the low-light training image used for training, A enhance (u, v) represents the amplitude of the enhanced image, D icd (A input (u,v),A enhance (u,v)) represents the magnitude difference of Chebyshev distance; Construct the phase similarity loss function as shown below: L phase＝ 1-SSIM(P input (u,v),P enhance (u,v)); Among them, SSIM(P input (u,v),P enhance (u,v)) represents the phase structure similarity between the low-light training image and the enhanced image in the frequency domain.

8. An end-to-end low-light image detection device based on frequency domain enhancement, characterized in that: include: Transformation module, frequency domain processing module, enhancement module and training module; The transformation module is configured to input the low-light training image for training into the frequency domain enhanced target detection network to be trained, and use the Fourier enhancement sub-network in the frequency domain enhanced target detection network to perform fast Fourier transformation on the low-light training image to obtain a frequency domain feature map; The frequency domain processing module is configured to perform complex convolution on the frequency domain feature map, and continuously perform multiple frequency domain processing on the result of the complex convolution to obtain a spectrum feature map, perform dimension reduction on the spectrum feature map to obtain a first frequency domain conversion map, and use a preset activation function to constrain the range of the first frequency domain conversion map to obtain a second frequency domain conversion map; The enhancement module is configured to determine the amplitude and phase of the frequency domain feature map, amplify the amplitude of the frequency domain feature map by dividing the amplitude of the frequency domain feature map by the second frequency domain conversion map, and perform an inverse fast Fourier transform using the phase and the amplified amplitude to obtain an enhanced image; The training module is configured to input the low-light training image and the enhanced image into the target detection subnetwork in the frequency domain enhanced target detection network, perform end-to-end training on the frequency domain enhanced target detection network using a preset joint loss function in the target detection subnetwork, and use the trained frequency domain enhanced target detection network to identify the target body in the low-light image to be detected.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executed by the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method according to any one of claims 1 to 7.

Images