A high-precision integrated imaging method for 3D salient target detection with texture features

Abstract

This invention proposes a high-precision integrated imaging 3D salient target detection method with texture features. This method comprises four processes: acquisition of four-dimensional light field data, high-precision salient target detection using a micro-image array, texture feature compensation of the salient micro-image array, and 3D reconstruction of the high-precision salient micro-image array with texture features. The method generates a micro-image array from the acquired four-dimensional light field data, using it as input for high-precision salient target detection. Through recombination and folding operations and cyclic encoding and decoding, it fully mines cross-regional global salient features to generate a high-precision salient micro-image array. Then, using a hierarchical texture feature compensation method, it restores and generates texture features for both regional and edge salient micro-image arrays. Finally, it uses a lens array to perform 3D reconstruction of the high-precision salient micro-image array with texture features, effectively simplifying subsequent computer vision tasks and providing fine-grained, colorful 3D display of salient targets.

Description

I. Technical Field

[0001] This invention relates to the field of salient target detection in computer vision tasks, and more specifically, to a high-precision integrated imaging 3D salient target detection method with texture features. II. Background Technology

[0002] Integrated imaging is a naked-eye 3D stereoscopic imaging technology characterized by full parallax and full-color display. It utilizes lens arrays to efficiently store and reconstruct 3D image data, increasing information capacity and display dimensionality. With the development of science and technology, the rich depth information and realistic texture details contained in integrated imaging 3D images provide strong support for medical image visualization analysis and autonomous driving environmental perception. However, how to quickly extract key and effective 3D information from this massive amount of data is a crucial issue in integrated imaging image analysis and reconstruction. 3D salient object detection methods can identify the most eye-catching objects from 3D images, achieving the detection and segmentation of visually unique image regions, greatly simplifying subsequent computer vision tasks.

[0003] In research on salient object detection, traditional methods lack depth information, making it impossible to reconstruct 3D objects and hindering spatial context understanding. Combining salient object detection with ensemble imaging has been explored as a preliminary approach to address this limitation, aiming to reconstruct 3D salient objects. However, this method still faces significant challenges such as limited resolution of individual micro-images and insufficient integration of global information, leading to critical issues like lost details, false positives, and false negatives. Furthermore, existing 3D salient object detection methods typically output only black-and-white binary images, failing to fully capture the rich and varied texture features of 3D salient object surfaces, thus limiting the ability to provide users with an immersive 3D experience. III. Summary of the Invention

[0004] This invention proposes a high-precision integrated imaging 3D salient target detection method with texture features. Its core lies in effectively integrating four-dimensional light field data to enhance the recognition accuracy of 3D salient targets and restore their true texture features. The method first samples the four-dimensional light field, using the obtained micro-image array as network input. Then, it segments the micro-image sequence, reassembles and folds the sequence, and performs n-fold cyclic encoding to achieve different recombination methods of the four-dimensional light field information of micro-images in different regions, fully extracting the salient features of the micro-image sequence. The results are then input in parallel to the decoding module to achieve multi-scale feature fusion and image resolution restoration, resulting in a high-precision salient micro-image array. Next, the salient micro-image array undergoes classification texture feature compensation processing, regional salient micro-image arrays are subjected to image thresholding processing, and a deep convolutional generative adversarial network is used to process the boundary salient micro-image array, generating true color texture features of the salient object boundaries. This outputs a high-precision salient micro-image array with texture features. Finally, a 3D image of the salient object is reconstructed using an integrated imaging display device.

[0005] The method comprises four processes: acquisition of four-dimensional light field data, high-precision salient target detection of micro-image arrays, texture feature compensation of salient micro-image arrays, and three-dimensional reconstruction of salient micro-image arrays with texture features. The specific process is detailed in the appendix. Figure 1 As shown.

[0006] The acquisition of the four-dimensional light field data can be achieved by establishing a real camera array to collect real four-dimensional light field information, or by establishing a virtual camera array in computer graphics software to acquire virtual four-dimensional light field information. Then, a micro-image array is generated using a pixel mapping algorithm. The micro-image array contains multi-angle information of the four-dimensional light field and rich geometric information of the scene. Comprehensive analysis of the micro-image array can effectively uncover multi-angle three-dimensional saliency relationships within different spatial ranges.

[0007] The high-precision salient target detection process of the micro-image array is shown in the attached figure. Figure 2 As shown, an M×N micro-image array is divided into micro-image sequences according to their dimensions. After reconstruction, a three-dimensional tensor is obtained, which is then divided into k×k adjacent micro-images to form a small rectangular patch. In the folding operation, a convolutional kernel of the same size as the patch, with a stride of s and padding of p, is used. The kernel is convolved with the corresponding elements within the patch. After passing through a flattening layer, a micro-image sequence with increased length is obtained. This sequence, along with a sinusoidal positional code, is embedded into the encoder as input. n attention heads are set, and after passing through a multi-head attention layer, multiple sets of query matrices Q, key matrices K, and value matrices V of the micro-image sequences at each level are obtained. The normalized attention weight A of the i-th head is obtained through dot product calculation, scaling factor, and activation function. i ,Right now:

[0008]

[0009] According to A i For the corresponding attention head V i The context vector at the corresponding position in the micro-image sequence is obtained by weighted summation, and then fully connected before being fed into the multilayer perceptron module, as shown in the attached diagram. Figure 3 As shown, after nonlinear transformation, the context vector is further used to extract high-level micro-image features in the hidden layer. After residual connections and feature-dimensional-based layer normalization, the encoder output is obtained. In the decoding module, collaborative upsampling is achieved using the decoder and inverse recombination folding operation. In the masked multi-head attention layer, the dimensions of the input and output features of the micro-image sequence are set, and independent attention weights are calculated for all positions of the micro-images within the standard set. In the encoder-decoder attention sub-layer, the image feature dimension is set to be the same as the input and output feature dimensions to facilitate scaling the dot product to calculate the correlation between micro-images across regions and to perform residual connection calculations between two consecutive micro-image sequence layers. In the multilayer perceptron and fully connected layer, the input features of the previous layer are upsampled. To preserve detail information, this layer fuses low-level features of micro-image sequences from encoders at different scales. By using inverse recombination folding and the cyclic stacking of the decoder, the image size is restored to the original size of the micro-image array. With m cyclic decoding modules and the same number of scaling factors r, the feature micro-image array is gradually restored to its original resolution, resulting in a high-precision salient micro-image array.

[0010] The texture feature compensation process for the significant micro-image array is as follows: Figure 4 As shown, the black-and-white salient micro-image array is divided into a region-salient micro-image array and an edge-salient micro-image array using a Gaussian filter, where each micro-image array contains M×N pixels. Image thresholding is performed on the region-salient micro-image array, where the texture features of the corresponding region in the original image f(x,y) are restored for the white regions, while the black regions are preserved. The resulting image g(x,y) is represented as:

[0011]

[0012] A deep convolutional generative adversarial network (GAN) is used to generate texture features for a boundary-salient micro-image array. The generator consists of transposed convolutional layers, defined as generating a color image with red, green, and blue channels. Random noise z and a grayscale boundary map of resolution M×N are used as inputs to the generator. The random noise z is transformed into a higher-dimensional spatial representation and concatenated with the grayscale boundary map to obtain an original feature map of size a×b×c. Upsampling is performed through transposed convolutional layers, and batch normalization and rectified linear unit (RCU) activation functions are used in each transposed convolutional layer. The final layer of the generator is superimposed with a hyperbolic tangent function, outputting a color image of size M×N×3. The color image output by the generator enters the discriminator, which downsamples through a series of convolutional layers and RCU activation functions to capture features at different scales. After normalization, a sigmoid function is used in the final layer to map the output to the (0,1) interval to obtain the total loss value and update the generator parameters, ultimately generating a boundary-salient micro-image array with texture features. This is fused with a region salient micro-image array having texture features using image addition to obtain a region salient micro-image array having texture features.

[0013] The three-dimensional reconstruction process of the salient micro-image array with texture features involves reconstructing the four-dimensional light field angle and spatial information of the micro-image array obtained in the third step using a lens array, thereby obtaining a three-dimensional salient object with texture features.

[0014] This invention proposes a high-precision integrated imaging 3D salient target detection method with texture features. Based on the rich spatial and angular information contained in micro-image arrays, a method for micro-image sequence reconstruction and folding is designed and combined with a Transformer encoder and decoder to fully exploit the salient cues of cross-regional micro-images within the micro-image array, achieving high-precision integrated imaging 3D salient target detection. Simultaneously, a hierarchical texture feature compensation method for salient micro-image arrays is proposed. Image thresholding and deep convolutional generative adversarial networks are used to restore the texture features of regional and boundary salient micro-image arrays, respectively, restoring the true details and color information of the salient 3D object surface. Therefore, the 3D salient object reconstructed using a lens array not only effectively simplifies subsequent computer vision tasks but also provides a finely detailed, colorful 3D display of salient targets. IV. Description of the attached drawings

[0015] Appendix Figure 1 This is a flowchart of a high-precision integrated imaging 3D salient target detection method with texture features proposed in this invention.

[0016] Appendix Figure 2 This is a flowchart of a high-precision salient target detection process using a micro-image array.

[0017] Appendix Figure 3 This is a schematic diagram of the structure of a multilayer perceptron in high-precision salient target detection.

[0018] Appendix Figure 4 The flowchart shows a hierarchical texture feature compensation method for salient micro-image arrays.

[0019] It should be understood that the above figures are only schematic and are not drawn to scale. V. Detailed Implementation Methods

[0020] The following detailed description of a typical embodiment of the high-precision integrated imaging 3D salient target detection method with texture features proposed in this invention further illustrates the invention. It is necessary to point out that the following embodiments are only used for further illustrative purposes and should not be construed as limiting the scope of protection of this invention. Any non-essential improvements and adjustments made to this invention by those skilled in the art based on the above description still fall within the scope of protection of this invention.

[0021] This invention proposes a high-precision integrated imaging 3D salient target detection method with texture features, specifically comprising four processes: acquisition of four-dimensional light field data, high-precision salient target detection of a micro-image array, texture feature compensation of the salient micro-image array, and 3D reconstruction of the salient micro-image array with texture features. The detailed process is attached. Figure 1 As shown.

[0022] The process of acquiring the four-dimensional light field data involves establishing a virtual camera array in computer graphics software to obtain virtual four-dimensional light field information, and then generating a micro-image array through a pixel mapping algorithm. The micro-image array contains multi-angle information of the four-dimensional light field and contains rich geometric information of the scene. Comprehensive analysis of the micro-image array can fully and effectively explore the multi-angle three-dimensional saliency relationships in different spatial ranges.

[0023] The high-precision salient target detection process of the micro-image array involves dividing a 2000×2000 micro-image array into a micro-image sequence based on the size of the micro-images. After reconstruction, a three-dimensional tensor is obtained from the micro-image sequence, which is then divided into 7×7 adjacent micro-images forming a small rectangular region patch. During the folding operation, a convolution kernel of the same size as the patch, with a stride of 4 and padding of 2, is used. This kernel is convolved with corresponding elements within the patch. The result is then passed through a flattening layer to obtain a micro-image sequence with increased length. This sequence, along with a sinusoidal positional code, is embedded into the encoder as input. Six attention heads are set, and after passing through a multi-head attention layer, multiple query matrices Q, key matrices K, and value matrices V are obtained for each level of the micro-image sequence. The normalized attention weight A of the i-th head is obtained through dot product calculation, scaling factor, and activation function. i ,Right now:

[0024] According to A iThe value V of the corresponding attention head i Weighted summation yields the context vector at the corresponding position in the micro-image sequence. This vector is then fully connected and fed into the multilayer perceptron module, as shown in the attached diagram. Figure 3 As shown, the context vector undergoes a nonlinear transformation and is then used to further extract high-level micro-image features in the hidden layer. After residual connections and feature-dimensional layer normalization, the encoder output is obtained. In the decoding module, collaborative upsampling is achieved using the decoder and the inverse recombination folding operation. In the masked multi-head attention layer, the dimensions of both the input and output features of the micro-image sequence are set to 768, and independent attention weights are calculated for all locations within the standard set of micro-images. In the encoder-decoder attention sub-layer, the image feature dimension is set to be the same as the input and output feature dimensions to facilitate scaling the dot product to calculate the correlation between micro-images across regions and to perform residual connection calculations between two consecutive micro-image sequence layers. In the multilayer perceptron and fully connected layer, the input features of the previous layer are upsampled. To preserve detail information, this layer fuses low-level features from micro-image sequences in encoders at different scales. By using reverse recombination folding and cyclic superposition of decoders, the image size is restored to the original size of the micro-image array. Three cyclic decoding modules and the same number of scaling factors r (8, 4, and 1 respectively) are set so that the output feature maps are upsampled to a fraction of the original micro-image array size. 1. Gradually restore the feature micro-image array to its original resolution to obtain a high-precision salient micro-image array.

[0025] The texture feature compensation process for the significant micro-image array is as follows: Figure 4 As shown, the black-and-white salient micro-image array is divided into a region-salient micro-image array and an edge-salient micro-image array using a Gaussian filter, with each micro-image array containing 2000×2000 pixels. Image thresholding is performed on the region-salient micro-image array, where the texture features of the corresponding region in the original micro-image array f(x,y) are restored for white regions, while the black regions are preserved. The resulting image g(x,y) is represented as:

[0026]

[0027] A deep convolutional generative adversarial network (GAN) is used to generate texture features for a boundary-salient micro-image array. The generator consists of transposed convolutional layers, defined as generating a color image with red, green, and blue channels. Random noise z and a grayscale boundary map with a resolution of 2000×2000 are used as inputs to the generator. The random noise z is transformed into a higher-dimensional spatial representation and concatenated with the grayscale boundary map to obtain an original feature map of size 125×125×256. Upsampling is performed through transposed convolutional layers, and batch normalization and rectified linear unit (RCU) activation functions are used in each transposed convolutional layer. The final layer of the generator is superimposed with a hyperbolic tangent function, outputting a color image of size 2000×2000×3. The color image output by the generator enters the discriminator, which downsamples through a series of convolutional layers and RCU activation functions to capture features at different scales. After normalization, a sigmoid function is used in the final layer to map the output to the (0,1) interval to obtain the total loss value and update the generator parameters, ultimately generating a boundary-salient micro-image array with texture features. This is fused with a region salient micro-image array having texture features using image addition to obtain a region salient micro-image array having texture features.

[0028] The process of reconstructing a three-dimensional object with texture features using a salient micro-image array involves reconstructing the four-dimensional light field angle and spatial information of the micro-image array obtained in the third step using a lens array, thereby obtaining a three-dimensional salient object with texture features.

[0029] This invention proposes a high-precision integrated imaging 3D salient target detection method with texture features. Based on the rich spatial and angular information contained in micro-image arrays, a method for micro-image sequence reconstruction and folding is designed and combined with a Transformer encoder and decoder to fully exploit the salient cues of cross-regional micro-images within the micro-image array, achieving high-precision integrated imaging 3D salient target detection. Simultaneously, a hierarchical texture feature compensation method for salient micro-image arrays is proposed. Image thresholding and deep convolutional generative adversarial networks are used to restore the texture features of regional and boundary salient micro-image arrays, respectively, restoring the true details and color information of the salient 3D object surface. Therefore, the 3D salient object reconstructed using a lens array not only effectively simplifies subsequent computer vision tasks but also provides a finely detailed, colorful 3D display of salient targets.

Claims

1. A high-precision integrated imaging 3D salient target detection method with texture features, characterized in that, The method includes four processes: acquisition of four-dimensional light field data, high-precision salient target detection of micro-image array, texture feature compensation of salient micro-image array, and three-dimensional reconstruction of high-precision salient micro-image array with texture features. In the process of acquiring four-dimensional light field data, a real or virtual camera array is established to collect light field information from multiple angles and generate micro-image array through pixel mapping algorithm. In the process of high-precision salient target detection of micro-image array, the micro-image array is segmented to obtain micro-image sequence, and it is encoded and decoded n times in a loop to obtain a high-precision salient micro-image array; During the process of texture feature compensation of salient micro-image array, the texture features of the regional salient micro-image array and the boundary salient micro-image array are restored layer by layer to obtain a high-precision salient micro-image array with texture features. During the 3D reconstruction of the salient micro-image array with texture features, a lens array is used to obtain a colored 3D salient object, which fully shows the rich texture details on its surface, and realizes high-precision integrated imaging 3D salient target detection and reconstruction with texture features.

2. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that, A real camera array can be built to collect real four-dimensional light field information, or a virtual camera array can be built in computer graphics software to obtain virtual four-dimensional light field information. A micro-image array can be obtained by using a pixel mapping algorithm. The micro-image array contains multi-angle information of the four-dimensional light field. Performing salient target detection on it can fully capture the salient dependencies of micro-images over a large area.

3. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that, The micro-image sequence is obtained by segmenting the micro-image array according to the micro-image size. It contains multi-angle information of the four-dimensional light field and rich scene geometric information. After recombination, a three-dimensional tensor is obtained. Multiple adjacent micro-images in the three-dimensional tensor are taken to form a small rectangular region patch. In the folding operation, the patch is convolved with a convolution kernel of the same size. After the convolution result is passed through a flattening layer, a cross-regional micro-image sequence with increased length is obtained, which facilitates the mining of saliency relationships of micro-images in a larger spatial range during the encoding process.

4. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that, The cross-regional micro-image sequence and sinusoidal positional encoding are simultaneously embedded into the encoder as input. In the multi-head attention layer, n attention heads are set, resulting in multiple sets of query matrices Q, key matrices K, and value matrices V for each level of the micro-image sequence. Then, the attention weight A of the micro-image sequence of the i-th attention head is calculated. i Compare it with the value matrix V of the corresponding attention head. i After weighted summation, the context vector of the micro-image at the corresponding location is obtained. After fully connected and nonlinear transformation, the high-level semantic features of the micro-image are extracted in the hidden layer and used as an important basis for saliency prediction. After residual connection and layer normalization, the output is then processed.

5. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that, Within the decoding module, independent attention weights are assigned to micro-images at all locations of the standard salient micro-image array; in the encoder-decoder attention layer, the same feature dimensions as the input and output features are set to realize the correlation between micro-images across regions in the micro-image sequence and the calculation of two consecutive micro-image sequence layers; in the multilayer perceptron, upsampling of image features and multi-scale fusion of low-level features of micro-image sequences at each level of the encoder are realized simultaneously. By using the reverse recombination folding operation and the cyclic superposition of the decoder, the image size is restored to the same size as the original micro-image array, and finally a high-precision salient micro-image array is obtained.

6. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that, The salient micro-image array is divided into regional salient micro-image arrays and boundary salient micro-image arrays for hierarchical texture feature compensation. Both types of salient micro-image arrays contain M×N pixels. The regional salient micro-image array is subjected to image thresholding processing. The texture features of the original micro-image array f(x,y) are restored in the white areas, while the black areas are retained. The resulting regional salient micro-image array g(x,y) with texture features satisfies:

7. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that, A deep convolutional generative adversarial network (GAN) is used to generate texture features of salient object boundaries. The generator consists of transposed convolutional layers and can generate red, green, and blue three-channel color images. The generator transforms random noise z and concatenates it with a grayscale boundary salient map of resolution M×N to obtain an original feature map of size a×b×c. This map is then passed through iterative transposed convolutional layers, with batch normalization and rectified linear unit activation functions used in each layer. The last layer superimposes a hyperbolic tangent function to output a color image of size M×N×3. In the discriminator, this color image is passed through a series of convolutional layers and rectified linear unit activation functions to capture its features at different scales. After normalization and using a sigmoid function, the output is mapped to the (0,1) interval to obtain the total loss value. The generator parameters are continuously updated to generate an array of salient micro-images with texture features. This is fused with a region salient micro-image array having texture features using image addition to obtain a region salient micro-image array having texture features.

8. The high-precision integrated imaging 3D salient target detection method with texture features according to claim 1, characterized in that, The generated array of high-precision salient micro-images with texture features is used to reconstruct a true-color 3D salient object using a lens array.

Images