Integrated imaging 3D salient target detection method in low-light environment
By modifying the Retinex theory and the depth-width attention module of the Transformer model, the accuracy problem of salient target detection under low-light conditions was solved, high-precision three-dimensional object reconstruction was achieved, and the three-dimensional perception capability of light field technology in complex environments was improved.
Patent Information
- Application Number
- CN202510905559.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-02
- Publication Date
- 2025-10-17
AI Technical Summary
Existing salient target detection methods have difficulty achieving high-precision three-dimensional object reconstruction under low-light conditions, especially in underwater turbid medium scattering or low-light environments at night, where image contrast is reduced and noise interference is enhanced, affecting the accuracy of salient area segmentation and limiting the application of light field technology in three-dimensional perception in complex environments.
The Retinex theory is used to modify the saliency detection method. By restoring the light field information of the micro-image array and the depth-width attention module of the Transformer model, four-dimensional light field data acquisition and the saliency detection network are combined to achieve salient target detection in low-light environments.
It improves the accuracy of salient target detection under low-light conditions, enables high-precision 3D object reconstruction in complex environments, enhances the robustness of 3D reconstruction and semantic segmentation boundaries, and improves the reliability of visual tasks.
Smart Images
Figure CN120807787A_ABST
Abstract
Description
I. TECHNICAL FIELD
[0001] The present application relates to the field of salient object detection of computer vision tasks, and more particularly, to an integrated imaging 3D salient object detection method in low-light environment. II. BACKGROUND
[0002] As the core branch of light field technology, integrated imaging has a unique lens array structure that can capture four-dimensional light field data containing complete position information and depth clues, and realize three-dimensional information coding under non-coherent light through a two-dimensional micro image array. This light field expression based on micro image array provides the most promising naked-eye 3D display scheme, and more importantly, its multi-view correlation characteristic establishes a complete geometric constraint system for three-dimensional reconstruction. The parallax information recorded by each micro image can be directly converted into depth information in the spatial scene, and the spatial continuity of the target in the micro image array provides topological prior for three-dimensional reconstruction. Under this background, saliency detection on the micro image array has a dual significance: it can enhance the robustness of depth estimation through salient region positioning, and it can provide accurate semantic segmentation boundaries for target-oriented three-dimensional reconstruction, thereby improving the reliability of three-dimensional scene analysis in visual tasks.
[0003] In the related research in the field of salient object detection, traditional methods are difficult to realize the reconstruction of three-dimensional objects due to the lack of depth information, and this limitation significantly restricts the understanding of the spatial context. Although the combination of salient object detection and integrated imaging technology provides a preliminary exploration path to solve the above problems, the existing saliency detection methods for micro image arrays are limited to ideal lighting conditions. When facing the scattering of underwater turbid medium or the night low-illumination environment, the severely degraded light will cause problems such as reduced image contrast and enhanced noise interference, limiting the segmentation accuracy of saliency detection on the salient region. To our knowledge, the current saliency detection research for low-light 3D objects is still in the blank, which directly affects the three-dimensional perception application of light field technology in complex environments. III. SUMMARY
[0004] This paper proposes an integrated imaging 3D salient object detection method in low-light environments. Its core is to effectively restore and integrate four-dimensional light field data to enhance the recognition accuracy of 3D salient objects in low-light environments. The method first takes a micro-image array under low-light conditions and a micro-image array under normal-light conditions as input. Using Retinex theory, it extracts an illumination map from the normal-light micro-image array. After removing texture information, it couples the map with the low-light micro-image array to restore the light field information to its normal state. The processed micro-image array is then input into a saliency detection network. This network fully utilizes the light field information to efficiently model long-range dependencies between micro-images, accurately capturing and restoring light field details, thereby generating a salient micro-image array. Finally, the resulting salient micro-image array is used to achieve high-precision reconstruction of 3D salient objects in a spatial scene, completing the end-to-end process from low-light micro-image array input to salient object output.
[0005] The method includes four steps: acquiring four-dimensional light field data, restoring low-light light field information, performing saliency detection on the restored micro-image array, and three-dimensional reconstruction of the salient micro-image array. The specific process is shown in the attached figure. Figure 1 shown.
[0006] The acquisition process of the four-dimensional light field data can be achieved by deploying an actual camera array to collect real four-dimensional light field data, or by constructing a virtual camera array in computer graphics processing software to obtain virtual four-dimensional light field data, and then generating a micro-image array through pixel mapping. The micro-image array integrates the multi-angle visual information of the four-dimensional light field and the scene geometric characteristics.
[0007] The light field information recovery process in low light environment is as shown in the attached Figure 2 As shown in the figure, image A is first decomposed into a coupled illuminance map I and reflectance map R according to the Retinex image decomposition theory, namely, A = I × R. The traditional Retinex decomposition theory is then modified based on the acquisition process of the micro-image array, namely, A = (I + I') × R. A complementary illuminance map L is introduced, satisfying L × I' = 1, transforming the division paradigm in the traditional algorithm into multiplication. This results in the modified Retinex decomposition formula: A = R + I × L × R, where I × L × R represents the illumination noise. The complementary illuminance map L is obtained by taking the maximum value of the pixels in each channel of the image. The micro-image array obtained in low-light conditions is then decomposed into a pure reflectance map according to the modified Retinex theory, while the micro-image array obtained in normal-light conditions is decomposed into a pure illuminance map. The two are then coupled in the latent space to achieve a preliminary restoration of the micro-image array in low-light conditions. A diffusion model is then used to denoise and adjust the illumination of the preliminarily restored image. Ultimately, the restored micro-image array is obtained.
[0008] The significant target detection process is performed on the recovered micro image array, as shown in FIG. 2. Figure 3 The MxN micro image array is divided and formed into a micro image sequence according to the size of the micro image, and a three-dimensional tensor is obtained after reconstruction. Then, the three-dimensional tensor is divided into kxk adjacent micro images to form a patch. In the folding operation, a convolution kernel with the same size as the patch is used, with a step size of s and a padding of p. The corresponding elements in the patch are convolved to obtain a one-dimensional sequence after the flat layer is unfolded. The length of the micro image sequence is increased, and the cosine position encoding is embedded into the encoder as input. n attention heads are set, and a plurality of query matrices Q, key matrices K and value matrices V of the micro image sequence at each level are obtained through multi-head attention layers. The normalized attention weight A i , that is:
[0009]
[0010] In the application of the Transformer model to process the micro image sequence, the A i of each attention head V i is weighted and summed to obtain the context vector at the corresponding position in the micro image sequence. Then, the context vector is sent to the multi-layer perceptron module through the fully connected layer. At the same time, the A i generated by each attention layer is recorded, which will be used as the input of the breadth attention mechanism. The breadth attention module integrates the A i generated by each attention layer to provide local detection information for the model. In the decoding module, upsampling is realized through the decoder and the reverse reorganization folding operation. In the mask multi-head attention layer, the dimensions of the input and output features of the micro image sequence are preset, and the attention weights are calculated independently for all positions of the micro image in the standard set. The depth-breadth attention sublayer sets the image feature dimension to be consistent with the input and output feature dimensions, which facilitates the calculation of the correlation between cross-regional micro images through scaled dot product and the splicing calculation of breadth features and depth features in two different attention layers. In the multi-layer perceptron and fully connected layer, the input features of the previous layer are upsampled, and the low-level features of the micro image sequence in different scale encoders are fused to retain the image detail information. Finally, through the reverse reorganization folding and the cyclic superposition of the decoder, m cyclic decoding modules and the same number of scaling factors r are set to gradually restore the feature micro image array to the original resolution, and finally generate a high-precision significant micro image sequence.
[0011] The significant micro image array three-dimensional reconstruction process is shown in FIG. 3. Figure 4The micro image array obtained in the third step is subjected to texture information recovery, and then subjected to four-dimensional light field angle and spatial information reconstruction by means of a lens array, so as to obtain a three-dimensional salient object with texture characteristics.
[0012] The present application provides a low-light integrated imaging 3D salient object detection method. For the problem of low-light micro image array enhancement, the Retinex image decomposition theory is innovatively reconstructed based on its unique acquisition method, so as to effectively enhance the unpaired low-light micro image array. In view of the fact that the micro image array contains rich light field spatial and angular information, a micro image sequence reorganization folding method is designed to reduce the four-dimensional light field data to one dimension and deeply integrate with the Transformer model. Through the depth-breadth attention module, the saliency clues of global-local micro images in the micro image array are fully mined, so as to realize high-precision integrated imaging 3D salient object detection in a low-light environment. Finally, the three-dimensional salient object reconstructed by the lens array can not only output high-precision detection results in a complex scene, but also greatly simplify the subsequent computer vision task process. IV. BRIEF DESCRIPTION OF DRAWINGS
[0013] FIG. 1 is a flow chart of a low-light integrated imaging 3D salient object detection method according to the present application. Figure 1 FIG. 1 is a flow chart of a low-light integrated imaging 3D salient object detection method according to the present application.
[0014] FIG. 2 is a flow chart of low-light light field information recovery according to the present application. Figure 2 FIG. 2 is a flow chart of low-light light field information recovery according to the present application.
[0015] FIG. 3 is a schematic diagram of saliency detection on the recovered micro image array according to the present application. Figure 3 FIG. 3 is a schematic diagram of saliency detection on the recovered micro image array according to the present application.
[0016] FIG. 4 is a schematic diagram of three-dimensional reconstruction of the salient micro image array according to the present application. Figure 4 FIG. 4 is a schematic diagram of three-dimensional reconstruction of the salient micro image array according to the present application.
[0017] It should be understood that the above drawings are only schematic and not drawn to scale. V. DETAILED DESCRIPTION
[0018] A typical embodiment of the low-light integrated imaging 3D salient object detection method according to the present application will be described in detail below, and the present application will be further described in detail. It is necessary to point out here that the following embodiment is only used to further illustrate the present application and cannot be understood as limiting the protection scope of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application according to the above content of the present application, which still belongs to the protection scope of the present application.
[0019] The low-light integrated imaging 3D salient object detection method according to the present application specifically includes four processes of four-dimensional light field data acquisition, low-light light field information recovery, salient object detection on the recovered micro image array, and three-dimensional reconstruction of the salient micro image array, and the specific process is shown in FIG. 1.Figure 1 shown.
[0020] The acquisition process of the four-dimensional light field data can be achieved by deploying an actual camera array to collect real four-dimensional light field data, or by constructing a virtual camera array in computer graphics processing software to obtain virtual four-dimensional light field data, and then generating a micro-image array through pixel mapping. The micro-image array integrates the multi-angle visual information of the four-dimensional light field and the scene geometric characteristics.
[0021] The light field information recovery process in low light environment is as shown in the attached Figure 2 As shown in the figure, based on the Retinex image decomposition theory, image P is first decomposed into a coupled illumination map I and reflectance map R, namely, P = I × R. The traditional Retinex decomposition theory is then modified based on the acquisition process of the micro-image array, namely, P = (I + I') × R. The complementary illumination map L is introduced to satisfy L × I' = 1, transforming the division paradigm in the traditional algorithm into multiplication, resulting in the modified Retinex decomposition formula: P = R + I × L × R, where I × L × R represents the illumination noise. The complementary illumination map L is obtained by taking the maximum value of the pixels in each channel of the image. The micro-image array obtained in low-light conditions is then decomposed into a pure reflectance map according to the modified Retinex theory, while the micro-image array obtained in normal-light conditions is decomposed into a pure illumination map. The two are then coupled in the latent space to complete the preliminary restoration of the micro-image array in low-light conditions. The Diffusion model is then used to denoise and adjust the illumination of the preliminarily restored image. Ultimately, a micro-image array with normal light field information is obtained.
[0022] The process of performing salient target detection on the restored micro-image array is as shown in the attached Figure 3 As shown. The micro-image array of size 2000×2000 is divided according to the size of the micro-image and forms a micro-image sequence. After reconstruction, the micro-image sequence is obtained as a three-dimensional tensor, which is then divided into 7×7 adjacent micro-images to form a small rectangular area (patch). In the folding operation, a convolution kernel of the same size as the patch is used, with a step size of 4 and a padding of 2. The result is convolved with the corresponding elements in the patch, and after being expanded into a one-dimensional sequence through a flattening layer, a micro-image sequence of increased length is obtained. It is embedded into the encoder together with the cosine position code as input, and 6 attention heads are set. After the multi-head attention layer, multiple groups of query matrices Q, key matrices K and value matrices V of the micro-image sequences at each level are obtained. The normalized attention weight A of the i-th head is obtained through dot product calculation, scaling factor and activation function i ,Right now:
[0023]
[0024] In the application of the Transformer model to process the micro image sequence, first, A i The corresponding attention head V i is weighted and summed to obtain the context vector of the corresponding position in the micro image sequence. Then, the context vector is sent to the multilayer perceptron module after passing through the fully connected layer. At the same time, A i generated by each attention layer is recorded. These data will be used as the input of the breadth attention mechanism. The breadth attention module integrates A i generated by each attention layer to provide local detection information for the model. In the decoding module, upsampling is realized through the decoder and the reverse recombination folding operation. In the mask multi-head attention layer, the dimensions of the input and output features of the micro image sequence are preset, and the attention weights of all positions of the micro image in the standard set are calculated independently. The depth-breadth attention sublayer sets the image feature dimension to be consistent with the input and output feature dimensions, which facilitates the calculation of the correlation of cross-regional micro images through scaled dot product and the splicing calculation of breadth features and depth features in two different attention layers. In the multilayer perceptron and the fully connected layer, the input features of the previous layer are upsampled, and the low-level features of the micro image sequence in different scale encoders are fused to retain image detail information. Finally, by means of the reverse recombination folding and the cyclic superposition of the decoder, 3 cyclic decoding modules and the same number of scale factors r are set, which are 8, 4 and 1, respectively, so that the output feature map is upsampled to 1 of the original micro image array size, gradually restoring the feature micro image array to the original resolution, and finally generating a high-precision salient micro image array.
[0025] In the salient micro image array three-dimensional reconstruction process, the micro image array obtained in the third step is subjected to texture information recovery, and then the four-dimensional light field angle and spatial information are reconstructed by means of the lens array to obtain a three-dimensional salient object with texture characteristics.
[0026] The application provides a low-light integrated imaging 3D salient object detection method. For the problem of low-light micro image array enhancement, the Retinex image decomposition theory is innovatively reconstructed based on the unique acquisition mode of the micro image array, and the non-paired low-light micro image array is effectively enhanced. Since the micro image array contains rich light field spatial and angular information, a micro image sequence recombination folding method is designed to reduce the four-dimensional light field data to one dimension and deeply integrate with the Transformer model. Through the depth-breadth attention module, the saliency clues of global-local micro images in the micro image array are fully mined to realize high-precision integrated imaging 3D salient object detection in a low-light environment. Finally, the three-dimensional salient object reconstructed by the lens array can not only output high-precision detection results in a complex scene, but also greatly simplify the subsequent computer vision task process.
Claims
1. A method for detecting 3D salient objects using integrated imaging in low-light environments is proposed. Its core is to effectively restore and integrate four-dimensional light field data to enhance the recognition accuracy of 3D salient objects in low-light environments. The method first takes a microimage array under low-light conditions and a microimage array under normal-light conditions as input. Using Retinex theory, the illumination map is extracted from the normal-light microimage array. After removing texture information, the image is coupled with the low-light microimage array to restore the light field information to a normal state. Subsequently, the processed microimage array is input into a saliency detection network. The network fully utilizes the light field information to efficiently model long-range dependencies between microimages, accurately capture and restore light field details, and thus obtain a salient microimage array. Finally, the obtained salient microimage array is used to achieve high-precision reconstruction of salient three-dimensional objects in a spatial scene, completing an end-to-end process from low-light microimage array input to salient object output. The method comprises four steps: acquiring four-dimensional light field data, restoring low-light light field information, performing saliency detection on the restored microimage array, and finally reconstructing the salient microimage array in three dimensions.
2. The method for detecting 3D salient objects using integrated imaging in a low-light environment according to claim 1, wherein: A real camera array can be established to collect real four-dimensional light field information, or a virtual camera array can be established in computer graphics software to obtain virtual four-dimensional light field information, and a micro-image array can be obtained using a pixel mapping algorithm. The micro-image array contains multi-angle information of the four-dimensional light field, and performing salient target detection on it can fully capture the saliency dependencies of micro-images over a larger range.
3. The method for detecting 3D salient objects using integrated imaging in a low-light environment according to claim 1, wherein: In the process of light field information recovery in low-light environments, according to the Retinex image decomposition theory, the image P is decomposed into P = I × R, and then corrected to P = (I + I') × R based on the micro-image array acquisition process. The complementary illumination map L is introduced, which is obtained by the maximum value of the pixels in each channel of the image and satisfies L × I' = 1, and the correction formula P = R + I × L × R is obtained; then the micro-image arrays under low light and normal illumination are decomposed into reflectance maps and illumination maps, respectively, and preliminary recovery is completed by coupling in the latent space. Finally, the Diffusion model is used to denoise and adjust the illumination to obtain a micro-image array with normal light field information.
4. The method for detecting 3D salient objects using integrated imaging in a low-light environment according to claim 1, wherein: The micro-image sequence is obtained by segmenting the micro-image array according to the micro-image size. It contains multi-angle information of the four-dimensional light field and rich scene geometry information. After reorganization, it is obtained into a three-dimensional tensor. Multiple adjacent micro-images in the three-dimensional tensor are taken to form a small rectangular area (patch). In the flattening operation, the patch is convolved with a convolution kernel of the same size. After the convolution result passes through the flattening layer, a cross-region micro-image sequence with increased length is obtained, which facilitates the mining of micro-image saliency relationships within a larger spatial range during the encoding process.
5. The method for detecting 3D salient objects using integrated imaging in a low-light environment according to claim 1, wherein: The cross-region micro-image sequence and the sinusoidal position code are simultaneously embedded into the encoder as input. In the multi-head attention layer, n attention heads are set to obtain multiple sets of query matrices Q, key matrices K and value matrices V of the micro-image sequences at each level, and then the attention weight A of the i-th head micro-image sequence is calculated. i , and compare it with the value matrix V of the corresponding attention head i After weighted summation, the context vector of the micro-image at the corresponding position is obtained. This vector can be used to extract the high-level semantic features of the micro-image in the hidden layer after full connection and nonlinear transformation, and serve as an important basis for saliency prediction.
6. The method for detecting 3D salient objects using integrated imaging in a low-light environment according to claim 1, wherein: In the decoding module, independent attention weights are assigned to the microimages at all positions in the standard salient microimage array. In the depth-width attention layer, the feature dimension is set to be the same as the width feature and the depth feature to realize the correlation between cross-regional microimages in the microimage sequence and the calculation of two consecutive microimage sequence layers, thereby improving the local accuracy of model detection. In the multi-layer perceptron, upsampling of image features and efficient fusion of depth-width features are simultaneously achieved. The image size is restored to the same size as the original microimage array by using the reverse recombination and folding operation and the cyclic superposition of the decoder, ultimately obtaining a salient microimage array of three-dimensional objects in a low-light environment.
7. The method for detecting 3D salient objects using integrated imaging in a low-light environment according to claim 1, wherein: The salient micro-image array generated in low-light environment is used to reconstruct the 3D salient object with normal texture information, contour information and spatial information using a lens array.